Googlebot is getting quicker

[twthumb]

I just was watching my logs for maybe something interesting and saw google coming by. I always thought google wouldn't flood a server with request:

66.249.66.211 - - [11/Aug/2005:21:22:48 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:22:53 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:22:55 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:22:56 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:03 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:06 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:07 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:12 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:14 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:18 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:21 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:23 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:25 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:42 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:44 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:46 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:50 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:52 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:53 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:56 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:58 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:00 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:05 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:06 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:15 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:17 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:18 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:18 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:18 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:18 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:27 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:28 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:33 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:36 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:37 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:39 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:42 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:44 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:46 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:49 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:51 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:52 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:55 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:57 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:59 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:25:08 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:25:10 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:25:11 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:25:16 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Of couse you can request pages a lot faster, but this is kind of strange of google, sometimes more than 4 request a second.

[cj]

i noticed that google spiders my sites, 3 times aday, to yahoo's 320.... And ya'll heard of google's new site map submit... Make a site map and then submit it to google, got my pages in the search engine, but they suck, only showing the URL and title or description... So another words... don't submit a site map...

Yahoo is kinda using the same idea, by saving a text file to your server called "urllist.txt", you can use a site spider, and spider your site your self. save it to the text file then submit your text file to yahoo....

when yahoo adds the url, they just don't like the url, but url, title and description...

another thing i noticed about google.com, they don't like tags.. like default.asp?id=1000

you can get your pages in with tags... but it takes alot longer... fuck it sucks...

i think google is going down hill and yahoo is going to fuck them up :o)

[Erick G]

Google is buzy spending bazillions of dollars from IPO

[twthumb]

Hmm, i never made a sitemap, i might take a look at it, thanks for the tip!

I also thought that google didn't like query strings (?id=x&do=that) but for some reason he seems to like my query strings, i can't find any reason why. For example the above request all hit a page which has a url with a query string. Google is a strange thing

[cj]

google hates query strings... But after while they seem to add them... but it takes months & months... Don't submit a site map!!!

This is what your pages will look like, Exmaple:
http://www.google.com/search?hl=en&q...jspornsite.com

No title or description just url address...

Use Yahoo's urllist.txt, its up to you, let me know if you have better luck than my self.

[twthumb]

Originally Posted by cj

google hates query strings... But after while they seem to add them... but it takes months & months... Don't submit a site map!!!

This is what your pages will look like, Exmaple:
http://www.google.com/search?hl=en&q...jspornsite.com

Ooh, that sucks big time. You convinced me to never make a site map for a new site.
I hope google picks up your site soon.

[cj]

thanks alot, but i don't think so... site maps fuck you... so pass the word...

oh yeah... wanna exchange links? :o)

cj@deluxeelectronics.com

[Ace-imlive]

Originally Posted by cj

thanks alot, but i don't think so... site maps fuck you... so pass the word...

oh yeah... wanna exchange links? :o)

cj@deluxeelectronics.com

google is slower

but just create a HTML simple sitemap

versus the xml they ask for or the .gz

then the bot grabs it no problem

[cj]

my site was up 6-22-05 and i already got 2,700 pages in yahoo.com. use yahoo, fuck google...

Don't rely on g, it takes months from them to add pages, or maybe they just hate my site...