You can run adult off an old P3 with a gig of ram and still get good performance, it's all in the tuning
That's why if you get a dedicated server, you also want it managed by capable sysadmins that actually know more than just "ugh reboot server, ugh install patch"
On a side note, if you don't use any fancy .htaccess files and have root access to your box, consider installing lighttpd instead of Apache. It's more memory efficient, and optimised to serve static content, but also works -very- well with PHP (by way of FastCGI).