There has to be something bad with your web server itself. Are you using apache?
I use apache on a centos dedicated box. I use op_cache now because it solved my problem with memcache and apc.
Varnish wasn't doing to good of a job with static data with oxwall so i disabled it.
I find it odd that your live site and test site on the same machine is having the issue but every one else environment cant reproduce it. It has to be something with your web server itself.