# /robots.txt file for http://webcrawler.com/ # mail webmaster@webcrawler.com for constructive criticism # http://www.robotstxt.org/wc/faq.html#use # http://www.yooda.com/info/article/robots_txt/ User-agent:* Disallow:/httpsdoc/