SEWATCH: The Big List of Web Robots

Who sent that web robot, and what is it doing crawling around on your server? Identify and track robots with this list of hundreds of active crawlers, link checkers and other cybercritters.

"In reality, crawlers are relatively simple programs, though they have the power to bring a web site to a standstill. They can also automatically and rapidly fetch material that a site owner may not want anyone to see. For this reason, most crawlers (also called "robots") abide by the "robots exclusion protocol," an informal set of rules that constrains their behavior." Who sent that web robot, and what is it doing crawling around on your server? Identify and track robots with this list of hundreds of active crawlers, link checkers and other cybercritters.