Greenguy's Board - View Single Post - robots.txt vs. meta name="ROBOTS" content="ALL, INDEX, FOLLOW

uberduber · 2007-11-07, 10:57 PM

I have seen many sites with either one or the other.. some with both. I am curious as to which is better and if the sites with no robot.txt is doing fine why is it needed(other than to prohibit bots from indexing certain areas). And if the site with meta robot text is doin fine why the robot txt?

2007-11-07, 10:57 PM	#1
uberduber Internet! Is that thing still around? Join Date: Sep 2006 Posts: 9	robots.txt vs. meta name="ROBOTS" content="ALL, INDEX, FOLLOW I have seen many sites with either one or the other.. some with both. I am curious as to which is better and if the sites with no robot.txt is doing fine why is it needed(other than to prohibit bots from indexing certain areas). And if the site with meta robot text is doin fine why the robot txt?