robots txt to prevent crawling of freesites

[Preacher]

Originally Posted by Jel

...I was running a link checker and was getting flags with this message:
"The link was not checked due to robots exclusion rules. Check the link manually."
Hence me looking at the root robots file.

This is actually more of the point that I was interested in. There is a submitter out there who is disallowing any bot that hits his free-sites, I believe through htaccess.

Therefore whenever my bot ran across his domains all of his free-sites would get flagged and pulled as unavailable and I'd have to re-add them manually. This was incredibly annoying.

I actually did this for a while but finally got tired of it and just left his sites in an error/delisted status. He no longer submits to me, but I still see his name pop-up from time to time.