 |
|
2013-08-21, 06:03 AM
|
#5 |
|
Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button
Join Date: Nov 2003
Posts: 157
|
If I am reading your post correctly, you're saying that the referrer was from robots.txt
All respectable spiders check robots.txt first before spidering your site, I doubt they'd send that as the referrer - but maybe that's what happened? A spider checked robots.txt, used the last page it accessed as its referrer and hence what you saw in your logs.
__________________
To alcohol! The cause of, and solution to, all of life’s problems |
|
|
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
Threaded Mode