Skippable Details

All this traffic is round robin'd to 2 apache2 servers (each is 2x 2.86 quad xeons w/ 4GB ram) and one backend db server of similar spec.

Over the years familiar problems rear there heads, the mysql "Too Many Connections" error, Apache running like a dog under stress etc.

This weekend I got the chance to rebuild the servers from scratch, and a chance to optimize everything - works very very well now!

Easy Optimization - Apache2

KeepAliveTimeout: Number of seconds to wait for the next request from the same client on the same connection.

Translate this to "Number of seconds to lock up a worker thread just incase another request comes in from the same client on the same connection" - in short a lot of browsers will request the page, keep the connection alive and use it to request images, css etc.

The default is 15 seconds; in my experience about 40-50% of connections will use KeepAlive, so if you have enough ram to support 500 requests before going in to swap memory, and you get 500 requests a second, by the time 3 seconds are past you've got a load of workers with RAM assigned doing nothing and all the hard work is getting done using swap memory instead, from this poitn on basically all the hard work is getting done using swap rather than ram, ya?

Thus, the simple fix is to knock KeepAliveTimeout right down - some people recommend 2 seconds, my personal preference is 4 seconds - works a charm for me :)

To analyse this yourself just check the server-status provided by apache (normally only viewable from localhost, easy way around this is to <?php echo file_get_contents('http://localhost/server-status'); ?>

Easy Optimization - MySQL Server

Other than the normal cnf values you should be optimizing [ key_buffer, sort_buffer_size, max_allowed_packet, thread_stack, thread_cache_size, max_connections, table_cache, query_cache_limit, query_cache_size ] - there is one more that is very important; namely wait_timeout.

wait_timeout - The number of seconds the server waits for activity on a noninteractive connection before closing it.

By default this value is very high, but often you get connections that just aren't closed properly, especially in the PHP apache world - if you ever hit that "Too many connections" error or see a lot of connections open for prolonged periods of time then this could / should be addressed.

Often you will see connections sitting idle for 10-15 seconds, for no good reason. Best to address it I'd say; and thus we lower wait_timeout to 5 seconds.

Connections are cleared faster, more resources free for other connections and all round its good. Remember one of mysql's strongest features is it's ability to open and close connections ultra-fast.

Summary

Honestly just check the your apache server status, see how many connections are open, how many of them are flagged K for keep alive then drop the keep alive time, take a note of processor usage / memory usage, drop the KeepAliveTimeout to 2, 4, 6, 8 and check it all again, you may be very suprised, it sure beats weeks of manual code optimizatios to make those scripts run faster.

Regards