Quote:
Originally Posted by uberduber
I have seen many sites with either one or the other.. some with both. I am curious as to which is better and if the sites with no robot.txt is doing fine why is it needed(other than to prohibit bots from indexing certain areas). And if the site with meta robot text is doin fine why the robot txt?
|
SE bots look for a file named robots.txt. You can draw your own conclusion from that.
In addition, the sites that have both are probably leaving secondary instructions for things to exclude on any particular pages.
2007-11-08, 02:50 AM
Threaded Mode