Google Bot Info request

[Allfetish]

Originally Posted by Halfdeck

True, linking to index.htm/html can cause a havoc.

Note: Don't rely solely on robots.txt since Google likes to ignore it in some cases.

What is the reason for this? Possibly the rewriting of the url in regards to duplication? e.g.

www.somedomain.com/dir/
www.somedomain.com/dir/index.html

This is interesting if I read it right as I know many linkslists prefer an index.html reciprical link, so as to help somewhat with discouraging doorway submissions.