Currently i am on a VPS that has 2gb ram, with a bunch of cache modules installed including xcache (you could also use APC or eaccelerator), gzip/deflate enabled, and output buffering on.
Below is the config I currently use and over the past week I have averaged a load average of 1.39 and memory usage average of 1.3GB.
my.cnf
query_cache_type=1
query_cache_limit=3M
query_cache_size=256M
max_connections=80
wait_timeout=150
table_cache=256M
tmp_table_size=128M
max_heap_table_size=128M
thread_cache=160M
sort_buffer_size=512K
read_buffer_size=512K
read_rnd_buffer_size=3M
myisam_sort_buffer_size=128M
max_allowed_packet=5M
http/apache settings
TraceEnable Off
FileETag None
StartServers 2
MinSpareServers 3
MaxSpareServers 5
ServerLimit 150
MaxRequestsPerChild 5000
KeepAlive On
KeepAliveTimeout 1