About the robots.txt file

LD · 2008-03-13, 09:37 AM

Thanks...the thing I could not figure out is if I need the need the meta tag in my header if I have the file in my domain. I am not sure if the meta tag is a substitute for the file, or if the file eliminates the need for the meta tag...or if having both is an issue.

Maj. Stress · 2008-03-13, 06:33 PM

Acorrding to this site the meta tag limiting in comparison to the actual robots.txt file. You can control the spidering of an entire domain from a few lines depending on what your needs are. You can tell bots not to spider /images folders or cgi-bin, etc.

You said you wanted everything spidered. If you look around at some of the lesser used search engines, you might change your mind. It's amazing some of the things that get indexed.

2008-03-13, 09:37 AM	#1
LD wtfwjd? Join Date: May 2007 Posts: 2,103	Thanks...the thing I could not figure out is if I need the need the meta tag in my header if I have the file in my domain. I am not sure if the meta tag is a substitute for the file, or if the file eliminates the need for the meta tag...or if having both is an issue. __________________ Artisteer Wordpress Theme Generator Create Custom Themes! My Little Network Last edited by LD; 2008-03-13 at 09:49 AM..

2008-03-13, 06:33 PM	#2
Maj. Stress Progress rarely comes in buckets, it normally comes in teaspoons Join Date: Jun 2005 Location: Dark Side Of Naboo Posts: 1,289	Acorrding to this site the meta tag limiting in comparison to the actual robots.txt file. You can control the spidering of an entire domain from a few lines depending on what your needs are. You can tell bots not to spider /images folders or cgi-bin, etc. You said you wanted everything spidered. If you look around at some of the lesser used search engines, you might change your mind. It's amazing some of the things that get indexed.