robot.txt

[JK]

Originally Posted by Mr Spock

If you disallow the spiders onto your mirrors isn't that the same effect as nofollowing the recip links on those mirrors - Great way to lose friends!

Disallowing the spiders completely, and getting pages indexed but not passing link juice to link lists are totally different.

Originally Posted by faxxaff

A lot of robots including Googlebot will do nothing but ignore a robots.txt file.

You may see "Googlebot" visiting pages you have disallowed in your robots.txt, I would hazard a guess that this is not really Google. It's quite easy to spoof the user-agent.