Quote:
Originally Posted by Mr Spock
If you disallow the spiders onto your mirrors isn't that the same effect as nofollowing the recip links on those mirrors - Great way to lose friends!
|
Disallowing the spiders completely, and getting pages indexed but not passing link juice to link lists are totally different.
Quote:
|
Originally Posted by faxxaff
A lot of robots including Googlebot will do nothing but ignore a robots.txt file.
|
You may see "Googlebot" visiting pages you have disallowed in your robots.txt, I would hazard a guess that this is not really Google. It's quite easy to spoof the user-agent.