About the robots.txt file

[LD]

Assuming I want everything crawled, should I just leave it out completely? Should I use this line in the header:

"<META NAME="ROBOTS" CONTENT="ALL">"

What's the best way to go about this when you want to get crawled?

[Maj. Stress]

One simple text file in the root of your domain is all you need. Use your favorite search engine to find robot.txt tutorial. It will eliminate some 404's from your stats too.

[LD]

Thanks...the thing I could not figure out is if I need the need the meta tag in my header if I have the file in my domain. I am not sure if the meta tag is a substitute for the file, or if the file eliminates the need for the meta tag...or if having both is an issue.

[Maj. Stress]

Acorrding to this site the meta tag limiting in comparison to the actual robots.txt file. You can control the spidering of an entire domain from a few lines depending on what your needs are. You can tell bots not to spider /images folders or cgi-bin, etc.

You said you wanted everything spidered. If you look around at some of the lesser used search engines, you might change your mind. It's amazing some of the things that get indexed.

[LD]

Excellent info at that site, thanks!

[Halfdeck]

"<META NAME="ROBOTS" CONTENT="ALL">"

is useless and unnecessary.

-----------------------------

Here's the difference between META ROBOTS tag and robots.txt.

Robots.txt tells Google not to crawl a URL. "Crawl" meaning don't retrieve content from the page.

META ROBOTS noindex tells Google don't show this page in Google's index. For that META tag to do its magic, a URL must be crawled (otherwise Google can't see the META tag).

As far is Google is concerned, there's something else to worry about. URLs blocked by robots.txt disallow accumulate PageRank. If you're having indexing issues, robots.txt is not a great way to handle it. META ROBOTS=noindex also leads to blocked URLs accumulating PageRank. On the other hand, META RObOTS=noindex,nofollow may be treated by Google like a dangling page.

[LD]

Thanks Halfdeck, good info if I can digest it..

The issue I am having is that my main index page is being indexed and crawled fine ("domain/index.php" ), but I'm having problems with my cat pages "domain/linklist/oral.php" and so forth. I submitted a site map, and G tells me my pages are indexed, yet when I type the url of most of my catetory pages in the G search box, I get nothing. They do show up in Yahoo and msn, and probably others...but no G.

Perhaps I should just wait a while..? I mean after they are indexed, should I allow for a certain amount of time before they show up in the search? They were not even indexed before I submitted the site map, and that was only a week or so ago.

The interesting thing about all this is my shaved pussy cat DOES show up, yet it gets very little traffic and I have no trades there,,,(?)

[Simon]

Originally Posted by LusciousDelight

The interesting thing about all this is my shaved pussy cat...

[stuveltje]

Stu----->> just spit her diet coke on her screen when she saw that cat picture........hahahahahaha to funny

[LD]

Would make a great avatar...

[Halfdeck]

when I type the url of most of my catetory pages in the G search box

Usually, its harder to get deeper pages indexed than the home page simply due to lack of PageRank. If you have a ton of outbound links on every page (especially the home page), your site gives away too much juice to other sites instead of pumping that juice to internal pages.

Perhaps I should just wait a while..? I mean after they are indexed, should I allow for a certain amount of time before they show up in the search?

Yeah, usually Google will get around to indexing pages sooner or later. How quickly your pages get picked depends mostly on PageRank, but it also depends on other factors, like how often you update your site.

[LD]

Thanks, Halfdeck...that gives me something to go on...