Greenguy's Board - View Single Post - robots.txt vs. meta name="ROBOTS" content="ALL, INDEX, FOLLOW

Sleep[Walker] · 2007-11-08, 05:16 AM

This two things can't be compared. Robots.txt file is used to disallow search engine bots to crawl the certain partitions of your website. And meta tag "robots" with content "all, index, follow" is useles, because ALL bots index permissible pages by default.
But if you want to exclude, for example, only one page of your website without robots.txt, you need to change CONTENT to "no index, no follow".

2007-11-08, 05:16 AM	#3
Sleep[Walker] A woman is like beer. They look good, they smell good, and you'd step over your own mother just to get one! Join Date: Oct 2007 Posts: 52	This two things can't be compared. Robots.txt file is used to disallow search engine bots to crawl the certain partitions of your website. And meta tag "robots" with content "all, index, follow" is useles, because ALL bots index permissible pages by default. But if you want to exclude, for example, only one page of your website without robots.txt, you need to change CONTENT to "no index, no follow". __________________ Please pardon for any possible mistakes in my English. Last edited by Sleep[Walker]; 2007-11-08 at 05:22 AM..