Linux Magazine: Throttling Your Web Server

By ServerWatch Staff (Send Email)
Posted Dec 26, 2000


"... as I put more and more pictures online, I started to notice some pretty creepy CPU loads. Worse than that, my ISP neighbors were also starting to complain. After investigation, I determined that I was getting hit by not-so-nice "spiders": Web programs that, given a few starting points, recursively (and rapidly) fetch the contents of many pages. I believe most of these to be people on fast data connections (like my current cable modem that brings the equivalent of 2 T-1's into my house for 0 per month, yes!) innocently asking their Web browser to download a whole area."

"So, rather than pull my pictures offline, I decided to implement a throttler. I didn't care as much about transfer bandwidth as I did CPU, so I chose to track recent CPU activity for each visitor. Of course, HTTP has no concept of a "session," so I took a very easy shortcut: tracking by IP address. Yes, I know I've ranted in discussion forums a lot about how an IP address is not a user. But, for the purpose of throttling it seemed the most expedient choice." ... I decided to implement a throttler. I didn't care as much about transfer bandwidth as I did CPU, so I chose to track recent CPU activity for each visitor. Of course, HTTP has no concept of a session, so I took a very easy shortcut: tracking by IP address.

"Once I put my throttler in place, no IP address is allowed to suck more than seven percent of my CPU over a period of 15 seconds. Once the CPU threshold is reached, any additional request is met with a 503 error (service unavailable), which, according to RFC2616 (the HTTP/1.1 specification), also allows me to give a "retry after" value of 15 seconds to advise the program that this was a temporary condition."

Page 1 of 1


Comment and Contribute

Your name/nickname

Your email

(Maximum characters: 1200). You have characters left.


 

 


Thanks for your registration, follow us on our social networks to keep up-to-date