 |
|
|
|
|
|
|
|
|
|
|||||||
 |
|
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
2007-11-07, 10:57 PM
|
#1 |
|
Internet! Is that thing still around?
Join Date: Sep 2006
Posts: 9
|
robots.txt vs. meta name="ROBOTS" content="ALL, INDEX, FOLLOW
I have seen many sites with either one or the other.. some with both. I am curious as to which is better and if the sites with no robot.txt is doing fine why is it needed(other than to prohibit bots from indexing certain areas). And if the site with meta robot text is doin fine why the robot txt?
|
|
|
|
2007-11-08, 02:50 AM
|
#2 | |
|
Progress rarely comes in buckets, it normally comes in teaspoons
Join Date: Jun 2005
Location: Dark Side Of Naboo
Posts: 1,289
|
Quote:
 In addition, the sites that have both are probably leaving secondary instructions for things to exclude on any particular pages.  |
|
|
|
|
|
2007-11-08, 05:16 AM
|
#3 |
|
A woman is like beer. They look good, they smell good, and you'd step over your own mother just to get one!
Join Date: Oct 2007
Posts: 52
|
This two things can't be compared. Robots.txt file is used to disallow search engine bots to crawl the certain partitions of your website. And meta tag "robots" with content "all, index, follow" is useles, because ALL bots index permissible pages by default.
But if you want to exclude, for example, only one page of your website without robots.txt, you need to change CONTENT to "no index, no follow".
__________________
Please pardon for any possible mistakes in my English. Last edited by Sleep[Walker]; 2007-11-08 at 05:22 AM.. |
|
|
|
|
2007-11-08, 11:04 AM
|
#4 |
|
Aw, Dad, you've done a lot of great things, but you're a very old man, and old people are useless
Join Date: Oct 2007
Location: GTA, Ontario
Posts: 26
|
This may sound hugely newb, but which kinds of things wouldn't you want a robot to index?
__________________
All My Cunning - check out my newbie webmaster blog |
|
|
|
|
2007-11-08, 01:03 PM
|
#5 |
|
Internet! Is that thing still around?
Join Date: Sep 2006
Posts: 9
|
well what about having neither robot.txt or the meta tag for follow index. it would seem that some have neither as well and are getting along great. just very curious. thanks
|
|
|
|
|
2007-11-08, 06:54 PM
|
#6 | ||
|
A woman is like beer. They look good, they smell good, and you'd step over your own mother just to get one!
Join Date: Oct 2007
Posts: 52
|
Quote:
Here's google's list. He is hidding files from himself. http://google.com/robots.txt Quote:
__________________
Please pardon for any possible mistakes in my English. |
||
|
|
|
|
2008-07-11, 05:05 PM
|
#7 | |
|
I want to set the record straight - I thought the cop was a prostitute
Join Date: Jul 2008
Location: Los Angeles
Posts: 297
|
Quote:
 |
|
|
|
|
|
|
|
Linear Mode
