I have a domain
http://www.torture.jp.net/ and checking my logs I found a hit on the index page from
http://www.torture.jp.net/robots.txt. Which would mean someone checked robots.txt before looking at the main page. Only problem with this is that I do
NOT have a robots.txt file on that domain!!! In other words, the hit came from a page that does not exist!
Is it possible that there is a "hidden" robots.txt file somewhere that I did not create, and does not show up on my FTP client? Because I am not only getting this strange referrer, but also Google refuses to index this domain, and I have got no hits from any search engine to this domain, ever. But all three things would make sense if there was a robots.txt file somewhere with a "no spider" rule set.