What the f.....?

Cleo · 2013-08-17, 07:57 AM

So you are saying that you are tortured by this conundrum.

ecchi · 2013-08-17, 08:13 AM

Quote:

Originally Posted by Cleo

So you are saying that you are tortured by this conundrum.

That and I'd like to see some search engine traffic!

sliiing · 2013-08-18, 04:56 AM

Quote:

Originally Posted by Cleo

So you are saying that you are tortured by this conundrum.

JK · 2013-08-21, 07:03 AM

If I am reading your post correctly, you're saying that the referrer was from robots.txt

All respectable spiders check robots.txt first before spidering your site, I doubt they'd send that as the referrer - but maybe that's what happened? A spider checked robots.txt, used the last page it accessed as its referrer and hence what you saw in your logs.

ecchi · 2013-08-21, 05:43 PM

Quote:

Originally Posted by JK

If I am reading your post correctly, you're saying that the referrer was from robots.txt.....

Yes, but the problem is that there is NO robots.txt on that domain! However all the search engines appear to have refused to list that domain, as though there was a robots.txt file with a "no list" rule! So I was asking if it is possible for there to be a hidden robots.txt file that does not show up on FTP?

JK · 2013-08-21, 07:06 PM

I doubt there can be a hidden robots.txt, unless there's a period before it - but then it wouldn't be robots.txt, it would be .robots.txt

Even if there is no robots.txt, a spider will still check for it. Does it show a 404 in your logs?

2013-08-17, 07:57 AM	#1
Cleo Subversive filth of the hedonistic decadent West Join Date: Mar 2003 Location: Southeast Florida Posts: 27,936	So you are saying that you are tortured by this conundrum. __________________ Free Rides on Uber and Lyft Uber Car: uberTzTerri Lyft Car: TZ896289

2013-08-21, 07:03 AM	#4
JK Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Nov 2003 Posts: 157	If I am reading your post correctly, you're saying that the referrer was from robots.txt All respectable spiders check robots.txt first before spidering your site, I doubt they'd send that as the referrer - but maybe that's what happened? A spider checked robots.txt, used the last page it accessed as its referrer and hence what you saw in your logs. __________________ To alcohol! The cause of, and solution to, all of life’s problems

2013-08-21, 07:06 PM	#6
JK Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Nov 2003 Posts: 157	I doubt there can be a hidden robots.txt, unless there's a period before it - but then it wouldn't be robots.txt, it would be .robots.txt Even if there is no robots.txt, a spider will still check for it. Does it show a 404 in your logs? __________________ To alcohol! The cause of, and solution to, all of life’s problems