Quote:
Originally Posted by Jel
...I was running a link checker and was getting flags with this message:
"The link was not checked due to robots exclusion rules. Check the link manually."
Hence me looking at the root robots file.
|
This is actually more of the point that I was interested in. There is a submitter out there who is disallowing any bot that hits his free-sites, I believe through htaccess.
Therefore whenever my bot ran across his domains
all of his free-sites would get flagged and pulled as unavailable and I'd have to re-add them manually. This was incredibly annoying.
I actually did this for a while but finally got tired of it and just left his sites in an error/delisted status. He no longer submits to me, but I still see his name pop-up from time to time.