Hot on HuffPost Tech:

See More Stories
AOL Tech

Tag: WEB CRAWLER

Daily Mail Hides Geekiest Job Listing Ever in Robots.txt File

Search engines typically dispatch "robots," or "spiders," to systematically crawl the Web for timely, pertinent and specific information. Programmers control the actions of those crawlers with robots.txt files, which command the spider to patrol certain websites and to index specific material. Inevitably, huge nerds will also hide quirky messages right in the heart of the geeky protocol. The ...