There seems to be a significant performance gain with Drupal running on PHP 5.5. The average memory usage on memory-hungry Views went down ~50% on a site I switched to PHP’s latest version. Seems the newly introduced OPcache works really well. If you’re not afraid of some warnings about the usage of (now) deprecated functions – go ahead and try it.
If your hosting company doesn’t offer PHP 5.5 it’s time to reconsider: give Uberspace a shot, they have 5.5 as default (and they also have an easy way to change to other versions, of course). Actually, even if your host does offer PHP 5.5, you should consider switching 😉