Quote:
|
Originally Posted by cd34
In robots.txt.....
|
Yes, that will work, so long as the spider bot actually follows the instructions. Google, Yahoo, etc probably do, but I'm not so sure about some of the lesser known SE's or what about ones that may not yet exist.