About the robots.txt file

Maj. Stress · 2008-03-13, 06:33 PM

Acorrding to this site the meta tag limiting in comparison to the actual robots.txt file. You can control the spidering of an entire domain from a few lines depending on what your needs are. You can tell bots not to spider /images folders or cgi-bin, etc.

You said you wanted everything spidered. If you look around at some of the lesser used search engines, you might change your mind. It's amazing some of the things that get indexed.

2008-03-13, 06:33 PM	#1
Maj. Stress Progress rarely comes in buckets, it normally comes in teaspoons Join Date: Jun 2005 Location: Dark Side Of Naboo Posts: 1,289	Acorrding to this site the meta tag limiting in comparison to the actual robots.txt file. You can control the spidering of an entire domain from a few lines depending on what your needs are. You can tell bots not to spider /images folders or cgi-bin, etc. You said you wanted everything spidered. If you look around at some of the lesser used search engines, you might change your mind. It's amazing some of the things that get indexed.