Greenguy's Board - View Single Post

JK · 2013-08-21, 07:03 AM

If I am reading your post correctly, you're saying that the referrer was from robots.txt

All respectable spiders check robots.txt first before spidering your site, I doubt they'd send that as the referrer - but maybe that's what happened? A spider checked robots.txt, used the last page it accessed as its referrer and hence what you saw in your logs.

2013-08-21, 07:03 AM	#5
JK Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Nov 2003 Posts: 157	If I am reading your post correctly, you're saying that the referrer was from robots.txt All respectable spiders check robots.txt first before spidering your site, I doubt they'd send that as the referrer - but maybe that's what happened? A spider checked robots.txt, used the last page it accessed as its referrer and hence what you saw in your logs. __________________ To alcohol! The cause of, and solution to, all of life’s problems