dcsimg

SEWATCH: The Big List of Web Robots


Download the authoritative guide: Data Center Guide: Optimizing Your Data Center Strategy

Download the authoritative guide: Cloud Computing: Using the Cloud for Competitive Advantage

Who sent that web robot, and what is it doing crawling around on your server? Identify and track robots with this list of hundreds of active crawlers, link checkers and other cybercritters.

"In reality, crawlers are relatively simple programs, though they have the power to bring a web site to a standstill. They can also automatically and rapidly fetch material that a site owner may not want anyone to see. For this reason, most crawlers (also called "robots") abide by the "robots exclusion protocol," an informal set of rules that constrains their behavior." Who sent that web robot, and what is it doing crawling around on your server? Identify and track robots with this list of hundreds of active crawlers, link checkers and other cybercritters.

This article was originally published on Oct 24, 2001
Page 1 of 1

Thanks for your registration, follow us on our social networks to keep up-to-date