About the robots.txt file

LD · 2008-03-13, 08:09 PM

Excellent info at that site, thanks!

Halfdeck · 2008-03-15, 01:21 PM

"<META NAME="ROBOTS" CONTENT="ALL">"

is useless and unnecessary.

-----------------------------

Here's the difference between META ROBOTS tag and robots.txt.

Robots.txt tells Google not to crawl a URL. "Crawl" meaning don't retrieve content from the page.

META ROBOTS noindex tells Google don't show this page in Google's index. For that META tag to do its magic, a URL must be crawled (otherwise Google can't see the META tag).

As far is Google is concerned, there's something else to worry about. URLs blocked by robots.txt disallow accumulate PageRank. If you're having indexing issues, robots.txt is not a great way to handle it. META ROBOTS=noindex also leads to blocked URLs accumulating PageRank. On the other hand, META RObOTS=noindex,nofollow may be treated by Google like a dangling page.

2008-03-13, 08:09 PM	#1
LD wtfwjd? Join Date: May 2007 Posts: 2,103	Excellent info at that site, thanks! __________________ Artisteer Wordpress Theme Generator Create Custom Themes! My Little Network

2008-03-15, 01:21 PM	#2
Halfdeck You can now put whatever you want in this space :) Join Date: Oct 2004 Location: New Haven, CT Posts: 985	"<META NAME="ROBOTS" CONTENT="ALL">" is useless and unnecessary. ----------------------------- Here's the difference between META ROBOTS tag and robots.txt. Robots.txt tells Google not to crawl a URL. "Crawl" meaning don't retrieve content from the page. META ROBOTS noindex tells Google don't show this page in Google's index. For that META tag to do its magic, a URL must be crawled (otherwise Google can't see the META tag). As far is Google is concerned, there's something else to worry about. URLs blocked by robots.txt disallow accumulate PageRank. If you're having indexing issues, robots.txt is not a great way to handle it. META ROBOTS=noindex also leads to blocked URLs accumulating PageRank. On the other hand, META RObOTS=noindex,nofollow may be treated by Google like a dangling page. __________________ Success is going from failure to failure without a loss of enthusiasm.