Downtime Today–Is My Own Fault
So I just had some downtime, basically caused because one of my sites was running a script which hunted for RSS feeds but didn’t time out. The feed URI was 404′ing, but every time a user hit the script they would start a new script which would run for the maximum execution time. Naturally, this continued until all of apache’s 512 connections were maxed out, essentially DOSing my own server. Pingdom says there were almost 3 hours of downtime:
Date Response time Uptime Downtime 5 1,257.524 ms 100.00% - 6 1,231.475 ms 100.00% - 7 1,341.041 ms 100.00% - 8 1,373.642 ms 100.00% - 9 1,228.253 ms 100.00% - 10 1,332.559 ms 100.00% - 11 1,218.832 ms 100.00% - 12 2,462.757 ms 87.84% 2h 46m
The response times also went through the roof. At this time, the load average was at least 50, way too high to serve requests quickly:
Things have settled down now, so there’s nothing to worry or fret over. I’m going to need to slowly migrate all of my RSS-grabbing scripts to a newer version with timeout, but for now Apache is behaving itself:
Current Time: Saturday, 12-May-2007 23:59:31 EDT
Restart Time: Saturday, 12-May-2007 23:21:44 EDT
Parent Server Generation: 1
Server uptime: 37 minutes 46 seconds
Total accesses: 31328 - Total Traffic: 2.1 GB
CPU Usage: u20.32 s4.48 cu0 cs0 - 1.09% CPU load
13.8 requests/sec - 0.9 MB/second - 68.8 kB/request
38 requests currently being processed, 13 idle workers
If anyone knows how I can increase my requests / s, let me know. I’m always wanting more speed.
This entry was posted on Saturday, May 12th, 2007 at 11:01 pm and is tagged with parent server generation, downtime today, maximum execution time, pingdom, cpu usage, cpu load, server uptime, response times, accesses, response time, timeout, current time, time out, 2h, ing, uri, scripts, apache, traffic. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback.
