Anyone know why Googlebot is doing this?

ecchi · 2015-09-27, 01:40 PM

When I started working on my unblockable pop up I initially sent it to an in between page with a META HTTP-EQUIV="refresh" to redirect it to it's final destination. That was up for only a few hours before I changed the target to a Perl CGI script. Apart from those few hours, when the page was a pop under there has never been, a link to it. However I did not bother to delete the page.

It had a random number to prevent caching, so it shows as:
bouncepage.shtml?X
where X can be any number between 1 and 1000.

Looking at my logs, I see a couple of IPs repeatedly trying the page with various values for X. Checking these IPs, they are Googlebot. Google has obviously picked up that link with several different numbers, and is looking at them every day.

Anyone have any idea why it would do this?

lezinterracial · 2015-09-29, 05:38 PM

That is weird. Maybe just the bad luck of Google bot seeing your site with the link for the few hours it was up. Or you have some script or job that runs that updates your sitemap.xml or some public stats log?

I found a site that checks whether Nursing care givers were still certified. Google showed three or four search results with the care givers SSN. I don't know why those results showed up in the search.

You might be able to block the one file in robots.txt, if you are careful.

ecchi · 2015-09-30, 03:59 AM

No site map on that site (it is not a well designed site) no public stats.

I think Google must have picked it up via the toolbar, because I have seen Google index my sites before a single link is up. And I did hit that page hard for a couple of hours while I perfected the script for the pop up. What I cannot see is why Google would want to keep checking it over.

It auto refreshes and picks one of five sites at random to auto refresh to. So the only thing I can think of is that Google still does not understand meta refresh (it used to see the content of the page refreshed to, as the content of the page doing the refresh) I thought they fixed that years ago, but if they have not, it may see each refresh as an update, and assume I have updated the page every time they visit, so think I update often, and keep re-visiting to keep re-indexing the updates!

But that does not make sense, because it would make their programmers idiots, and Google would not have become as big as it is if it's indexing made these stupid mistakes!

2015-09-27, 01:40 PM	#1
ecchi Banned Join Date: Oct 2003 Location: About to be evicted!!!! Posts: 4,082	Anyone know why Googlebot is doing this? When I started working on my unblockable pop up I initially sent it to an in between page with a META HTTP-EQUIV="refresh" to redirect it to it's final destination. That was up for only a few hours before I changed the target to a Perl CGI script. Apart from those few hours, when the page was a pop under there has never been, a link to it. However I did not bother to delete the page. It had a random number to prevent caching, so it shows as: bouncepage.shtml?X where X can be any number between 1 and 1000. Looking at my logs, I see a couple of IPs repeatedly trying the page with various values for X. Checking these IPs, they are Googlebot. Google has obviously picked up that link with several different numbers, and is looking at them every day. Anyone have any idea why it would do this?

2015-09-29, 05:38 PM	#2
lezinterracial Well you know boys, a nuclear reactor is a lot like women. You just have to read the manual and press the right button Join Date: Dec 2012 Posts: 152	That is weird. Maybe just the bad luck of Google bot seeing your site with the link for the few hours it was up. Or you have some script or job that runs that updates your sitemap.xml or some public stats log? I found a site that checks whether Nursing care givers were still certified. Google showed three or four search results with the care givers SSN. I don't know why those results showed up in the search. You might be able to block the one file in robots.txt, if you are careful. __________________ www.lezinterracial.com Best Free Cam Girls Last edited by lezinterracial; 2015-09-29 at 05:44 PM..

2015-09-30, 03:59 AM	#3
ecchi Banned Join Date: Oct 2003 Location: About to be evicted!!!! Posts: 4,082	No site map on that site (it is not a well designed site) no public stats. I think Google must have picked it up via the toolbar, because I have seen Google index my sites before a single link is up. And I did hit that page hard for a couple of hours while I perfected the script for the pop up. What I cannot see is why Google would want to keep checking it over. It auto refreshes and picks one of five sites at random to auto refresh to. So the only thing I can think of is that Google still does not understand meta refresh (it used to see the content of the page refreshed to, as the content of the page doing the refresh) I thought they fixed that years ago, but if they have not, it may see each refresh as an update, and assume I have updated the page every time they visit, so think I update often, and keep re-visiting to keep re-indexing the updates! But that does not make sense, because it would make their programmers idiots, and Google would not have become as big as it is if it's indexing made these stupid mistakes!