Greenguy's Board - View Single Post

JK · 2013-08-21, 07:06 PM

I doubt there can be a hidden robots.txt, unless there's a period before it - but then it wouldn't be robots.txt, it would be .robots.txt

Even if there is no robots.txt, a spider will still check for it. Does it show a 404 in your logs?

2013-08-21, 07:06 PM	#3
JK Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Nov 2003 Posts: 157	I doubt there can be a hidden robots.txt, unless there's a period before it - but then it wouldn't be robots.txt, it would be .robots.txt Even if there is no robots.txt, a spider will still check for it. Does it show a 404 in your logs? __________________ To alcohol! The cause of, and solution to, all of life’s problems