What the f.....?

ecchi · 2013-08-17, 06:33 AM

I have a domain http://www.torture.jp.net/ and checking my logs I found a hit on the index page from http://www.torture.jp.net/robots.txt. Which would mean someone checked robots.txt before looking at the main page. Only problem with this is that I do NOT have a robots.txt file on that domain!!! In other words, the hit came from a page that does not exist!

Is it possible that there is a "hidden" robots.txt file somewhere that I did not create, and does not show up on my FTP client? Because I am not only getting this strange referrer, but also Google refuses to index this domain, and I have got no hits from any search engine to this domain, ever. But all three things would make sense if there was a robots.txt file somewhere with a "no spider" rule set.

Cleo · 2013-08-17, 06:57 AM

So you are saying that you are tortured by this conundrum.

ecchi · 2013-08-17, 07:13 AM

Quote:

Originally Posted by Cleo

So you are saying that you are tortured by this conundrum.

That and I'd like to see some search engine traffic!

Greenguy · 2013-08-17, 07:58 AM

I'm no expert, but I'd assume that, if someone was looking for your robots.txt file, entered http://www.torture.jp.net/robots.txt into their browser, the .htaccess sends them to the index, since it's 404, hence a "referral" from a non-existing page.

ecchi · 2013-08-17, 02:33 PM

Thanks, that makes sense. Sadly it also means there is no mysterious robots.txt keeping the search engines out, and that they all just hate my site.

ecchi · 2013-08-17, 02:42 PM

Ahh, no that isn't it. Typing in http://www.torture.jp.net/robots.txt gives me "no referrer" and using the link in your post gives this thread as referrer.

sliiing · 2013-08-18, 03:56 AM

Quote:

Originally Posted by Cleo

So you are saying that you are tortured by this conundrum.

JK · 2013-08-21, 06:03 AM

If I am reading your post correctly, you're saying that the referrer was from robots.txt

All respectable spiders check robots.txt first before spidering your site, I doubt they'd send that as the referrer - but maybe that's what happened? A spider checked robots.txt, used the last page it accessed as its referrer and hence what you saw in your logs.

ecchi · 2013-08-21, 04:43 PM

Quote:

Originally Posted by JK

If I am reading your post correctly, you're saying that the referrer was from robots.txt.....

Yes, but the problem is that there is NO robots.txt on that domain! However all the search engines appear to have refused to list that domain, as though there was a robots.txt file with a "no list" rule! So I was asking if it is possible for there to be a hidden robots.txt file that does not show up on FTP?

JK · 2013-08-21, 06:06 PM

I doubt there can be a hidden robots.txt, unless there's a period before it - but then it wouldn't be robots.txt, it would be .robots.txt

Even if there is no robots.txt, a spider will still check for it. Does it show a 404 in your logs?

lezinterracial · 2013-08-25, 11:03 PM

I just checked on this site.
http://www.seositecheckup.com

Says your title is only "torture". I am thinking it might help to make a longer title.

It does say you use a Robots.txt file. Have you added one?

It also has a link to a robots.txt file check in webmaster tools. May help to check it.
https://support.google.com/webmaster...er/156449?rd=1

JAI-LING · 2013-08-26, 04:02 AM

Quote:

Originally Posted by ecchi

Thanks, that makes sense. Sadly it also means there is no mysterious robots.txt keeping the search engines out, and that they all just hate my site.

Probably "torture" is not a happy word for the SEs. In 2013 I would be worried to hit that link.

ecchi · 2013-08-29, 04:21 AM

Sorry, I read this and went to check out seositecheckup.com, then forgot to come back and thank all of you for your advice. So better late than never: Thanks.

_Richard_ · 2013-08-30, 03:50 PM

Quote:

Originally Posted by lezinterracial

I just checked on this site.
http://www.seositecheckup.com

Says your title is only "torture". I am thinking it might help to make a longer title.

It does say you use a Robots.txt file. Have you added one?

It also has a link to a robots.txt file check in webmaster tools. May help to check it.
https://support.google.com/webmaster...er/156449?rd=1

thanks for the link

CelineTotemCash · 2013-09-02, 05:57 AM

Quote:

Originally Posted by _Richard_

thanks for the link

+1

lezinterracial · 2013-09-09, 06:10 AM

I have been gone a while too. You are welcome.

2013-08-17, 06:33 AM	#1
ecchi Banned Join Date: Oct 2003 Location: About to be evicted!!!! Posts: 4,082	What the f.....? I have a domain http://www.torture.jp.net/ and checking my logs I found a hit on the index page from http://www.torture.jp.net/robots.txt. Which would mean someone checked robots.txt before looking at the main page. Only problem with this is that I do NOT have a robots.txt file on that domain!!! In other words, the hit came from a page that does not exist! Is it possible that there is a "hidden" robots.txt file somewhere that I did not create, and does not show up on my FTP client? Because I am not only getting this strange referrer, but also Google refuses to index this domain, and I have got no hits from any search engine to this domain, ever. But all three things would make sense if there was a robots.txt file somewhere with a "no spider" rule set.

2013-08-17, 06:57 AM	#2
Cleo Subversive filth of the hedonistic decadent West Join Date: Mar 2003 Location: Southeast Florida Posts: 27,936	So you are saying that you are tortured by this conundrum. __________________ Free Rides on Uber and Lyft Uber Car: uberTzTerri Lyft Car: TZ896289

2013-08-17, 07:58 AM	#4
Greenguy The Original Greenguy (Est'd 1996) & AVN HOF Member - I Crop Pics For Thumbs In My Sleep Join Date: Feb 2003 Location: Blasdell, NY (shithole suburb south of Buffalo) Posts: 41,758	I'm no expert, but I'd assume that, if someone was looking for your robots.txt file, entered http://www.torture.jp.net/robots.txt into their browser, the .htaccess sends them to the index, since it's 404, hence a "referral" from a non-existing page. __________________ Promote POV Porn Cash By Building & Submitting Galleries to the Porn Luv Network

2013-08-21, 06:03 AM	#8
JK Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Nov 2003 Posts: 157	If I am reading your post correctly, you're saying that the referrer was from robots.txt All respectable spiders check robots.txt first before spidering your site, I doubt they'd send that as the referrer - but maybe that's what happened? A spider checked robots.txt, used the last page it accessed as its referrer and hence what you saw in your logs. __________________ To alcohol! The cause of, and solution to, all of life’s problems

2013-08-21, 06:06 PM	#10
JK Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Nov 2003 Posts: 157	I doubt there can be a hidden robots.txt, unless there's a period before it - but then it wouldn't be robots.txt, it would be .robots.txt Even if there is no robots.txt, a spider will still check for it. Does it show a 404 in your logs? __________________ To alcohol! The cause of, and solution to, all of life’s problems

2013-08-17, 02:33 PM	#5
ecchi Banned Join Date: Oct 2003 Location: About to be evicted!!!! Posts: 4,082	Thanks, that makes sense. Sadly it also means there is no mysterious robots.txt keeping the search engines out, and that they all just hate my site.

2013-08-17, 02:42 PM	#6
ecchi Banned Join Date: Oct 2003 Location: About to be evicted!!!! Posts: 4,082	Ahh, no that isn't it. Typing in http://www.torture.jp.net/robots.txt gives me "no referrer" and using the link in your post gives this thread as referrer.

2013-08-25, 11:03 PM	#11
lezinterracial Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Dec 2012 Posts: 152	I just checked on this site. http://www.seositecheckup.com Says your title is only "torture". I am thinking it might help to make a longer title. It does say you use a Robots.txt file. Have you added one? It also has a link to a robots.txt file check in webmaster tools. May help to check it. https://support.google.com/webmaster...er/156449?rd=1 __________________ www.lezinterracial.com Best Free Cam Girls Last edited by lezinterracial; 2013-08-25 at 11:57 PM..

2013-08-29, 04:21 AM	#13
ecchi Banned Join Date: Oct 2003 Location: About to be evicted!!!! Posts: 4,082	Sorry, I read this and went to check out seositecheckup.com, then forgot to come back and thank all of you for your advice. So better late than never: Thanks.

2013-09-09, 06:10 AM	#16
lezinterracial Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Dec 2012 Posts: 152	I have been gone a while too. You are welcome. __________________ www.lezinterracial.com Best Free Cam Girls