Greenguy's Board - View Single Post

MadMax · 2005-06-12, 02:19 PM

Make sure you've got a robots.txt file in your main directory as well. I've noticed that when I slap up a site quickly and don't put that up right away Gbot is slower to take notice. Don't know if that's by design or coincidence, but it's still a good idea to have a robots.txt on every site

2005-06-12, 02:19 PM	#7
MadMax "Without evil there can be no good, so it must be good to be evil sometimes" ~ Satan Join Date: Aug 2004 Location: Motor City, baby, where carjacking was invented! Now GIMME THOSE SHOES! Posts: 2,385	Make sure you've got a robots.txt file in your main directory as well. I've noticed that when I slap up a site quickly and don't put that up right away Gbot is slower to take notice. Don't know if that's by design or coincidence, but it's still a good idea to have a robots.txt on every site