|
2013-08-17, 06:33 AM | #1 |
Banned
Join Date: Oct 2003
Location: About to be evicted!!!!
Posts: 4,082
|
What the f.....?
I have a domain http://www.torture.jp.net/ and checking my logs I found a hit on the index page from http://www.torture.jp.net/robots.txt. Which would mean someone checked robots.txt before looking at the main page. Only problem with this is that I do NOT have a robots.txt file on that domain!!! In other words, the hit came from a page that does not exist!
Is it possible that there is a "hidden" robots.txt file somewhere that I did not create, and does not show up on my FTP client? Because I am not only getting this strange referrer, but also Google refuses to index this domain, and I have got no hits from any search engine to this domain, ever. But all three things would make sense if there was a robots.txt file somewhere with a "no spider" rule set. |
2013-08-17, 06:57 AM | #2 |
Subversive filth of the hedonistic decadent West
Join Date: Mar 2003
Location: Southeast Florida
Posts: 27,936
|
So you are saying that you are tortured by this conundrum.
|
2013-08-17, 07:13 AM | #3 |
Banned
Join Date: Oct 2003
Location: About to be evicted!!!!
Posts: 4,082
|
|
2013-08-17, 07:58 AM | #4 |
The Original Greenguy (Est'd 1996) & AVN HOF Member - I Crop Pics For Thumbs In My Sleep
|
I'm no expert, but I'd assume that, if someone was looking for your robots.txt file, entered http://www.torture.jp.net/robots.txt into their browser, the .htaccess sends them to the index, since it's 404, hence a "referral" from a non-existing page.
|
2013-08-17, 02:33 PM | #5 |
Banned
Join Date: Oct 2003
Location: About to be evicted!!!!
Posts: 4,082
|
Thanks, that makes sense. Sadly it also means there is no mysterious robots.txt keeping the search engines out, and that they all just hate my site.
|
2013-08-17, 02:42 PM | #6 |
Banned
Join Date: Oct 2003
Location: About to be evicted!!!!
Posts: 4,082
|
Ahh, no that isn't it. Typing in http://www.torture.jp.net/robots.txt gives me "no referrer" and using the link in your post gives this thread as referrer.
|
2013-08-18, 03:56 AM | #7 |
Rock stars ... is there anything they don't know?
Join Date: Jul 2013
Posts: 18
|
|
2013-08-21, 06:03 AM | #8 |
Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button
Join Date: Nov 2003
Posts: 157
|
If I am reading your post correctly, you're saying that the referrer was from robots.txt
All respectable spiders check robots.txt first before spidering your site, I doubt they'd send that as the referrer - but maybe that's what happened? A spider checked robots.txt, used the last page it accessed as its referrer and hence what you saw in your logs.
__________________
To alcohol! The cause of, and solution to, all of life’s problems |
2013-08-21, 04:43 PM | #9 |
Banned
Join Date: Oct 2003
Location: About to be evicted!!!!
Posts: 4,082
|
Yes, but the problem is that there is NO robots.txt on that domain! However all the search engines appear to have refused to list that domain, as though there was a robots.txt file with a "no list" rule! So I was asking if it is possible for there to be a hidden robots.txt file that does not show up on FTP?
|
2013-08-21, 06:06 PM | #10 |
Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button
Join Date: Nov 2003
Posts: 157
|
I doubt there can be a hidden robots.txt, unless there's a period before it - but then it wouldn't be robots.txt, it would be .robots.txt
Even if there is no robots.txt, a spider will still check for it. Does it show a 404 in your logs?
__________________
To alcohol! The cause of, and solution to, all of life’s problems |
2013-08-25, 11:03 PM | #11 |
Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button
Join Date: Dec 2012
Posts: 152
|
I just checked on this site.
http://www.seositecheckup.com Says your title is only "torture". I am thinking it might help to make a longer title. It does say you use a Robots.txt file. Have you added one? It also has a link to a robots.txt file check in webmaster tools. May help to check it. https://support.google.com/webmaster...er/156449?rd=1 Last edited by lezinterracial; 2013-08-25 at 11:57 PM.. |
2013-08-26, 04:02 AM | #12 |
Currently in an Oppai Bar Getting Tittified!
Join Date: Apr 2008
Posts: 1,258
|
|
2013-08-29, 04:21 AM | #13 |
Banned
Join Date: Oct 2003
Location: About to be evicted!!!!
Posts: 4,082
|
Sorry, I read this and went to check out seositecheckup.com, then forgot to come back and thank all of you for your advice. So better late than never: Thanks.
|
2013-08-30, 03:50 PM | #14 | |
WHO IS FONZY!?! Don't they teach you anything at school?
Join Date: Feb 2010
Posts: 42
|
Quote:
|
|
2013-09-02, 05:57 AM | #15 |
Aw, Dad, you've done a lot of great things, but you're a very old man, and old people are useless
Join Date: Aug 2013
Posts: 20
|
__________________
Check out our Halloween Special $100 Bonus |
2013-09-09, 06:10 AM | #16 |
Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button
Join Date: Dec 2012
Posts: 152
|
I have been gone a while too. You are welcome.
|
|
|