Greenguy's Board

Greenguy's Board (http://www.greenguysboard.com/board/index.php)
-   Search Engines (http://www.greenguysboard.com/board/forumdisplay.php?f=13)
-   -   Googlebot is getting quicker (http://www.greenguysboard.com/board/showthread.php?t=22914)

twthumb 2005-08-11 03:32 PM
Googlebot is getting quicker
 
I just was watching my logs for maybe something interesting and saw google coming by. I always thought google wouldn't flood a server with request:

Quote:
66.249.66.211 - - [11/Aug/2005:21:22:48 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:22:53 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:22:55 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:22:56 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:03 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:06 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:07 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:12 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:14 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:18 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:21 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:23 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:25 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:42 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:44 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:46 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:50 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:52 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:53 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:56 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:23:58 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:00 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:05 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:06 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:15 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:17 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:18 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:18 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:18 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:18 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:27 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:28 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:33 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:36 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:37 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:39 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:42 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:44 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:46 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:49 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:51 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:52 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:55 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:57 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:24:59 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:25:08 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:25:10 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:25:11 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.211 - - [11/Aug/2005:21:25:16 +0200] "GET xxx "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Of couse you can request pages a lot faster, but this is kind of strange of google, sometimes more than 4 request a second.

cj 2005-08-11 09:51 PM
i noticed that google spiders my sites, 3 times aday, to yahoo's 320.... And ya'll heard of google's new site map submit... Make a site map and then submit it to google, got my pages in the search engine, but they suck, only showing the URL and title or description... So another words... don't submit a site map...

Yahoo is kinda using the same idea, by saving a text file to your server called "urllist.txt", you can use a site spider, and spider your site your self. save it to the text file then submit your text file to yahoo....

when yahoo adds the url, they just don't like the url, but url, title and description...

another thing i noticed about google.com, they don't like tags.. like default.asp?id=1000

you can get your pages in with tags... but it takes alot longer... fuck it sucks...

i think google is going down hill and yahoo is going to fuck them up :o)

Erick G 2005-08-11 10:15 PM
Google is buzy spending bazillions of dollars from IPO:)

twthumb 2005-08-12 01:28 PM
Hmm, i never made a sitemap, i might take a look at it, thanks for the tip!

I also thought that google didn't like query strings (?id=x&do=that) but for some reason he seems to like my query strings, i can't find any reason why. For example the above request all hit a page which has a url with a query string. Google is a strange thing ;)

cj 2005-08-13 07:09 PM
google hates query strings... But after while they seem to add them... but it takes months & months... Don't submit a site map!!!

This is what your pages will look like, Exmaple:
http://www.google.com/search?hl=en&q...jspornsite.com

No title or description just url address...

Use Yahoo's urllist.txt, its up to you, let me know if you have better luck than my self.

cj 2005-08-13 07:10 PM
my site was up 6-22-05 and i already got 2,700 pages in yahoo.com. use yahoo, fuck google...

Don't rely on g, it takes months from them to add pages, or maybe they just hate my site...

twthumb 2005-08-15 05:34 PM
Quote:
Originally Posted by cj
google hates query strings... But after while they seem to add them... but it takes months & months... Don't submit a site map!!!

This is what your pages will look like, Exmaple:
http://www.google.com/search?hl=en&q...jspornsite.com
Ooh, that sucks big time. You convinced me to never make a site map for a new site.
I hope google picks up your site soon.

cj 2005-08-16 06:21 AM
thanks alot, but i don't think so... site maps fuck you... so pass the word...

oh yeah... wanna exchange links? :o)

cj@deluxeelectronics.com

Ace-imlive 2005-08-29 06:06 AM
Quote:
Originally Posted by cj
thanks alot, but i don't think so... site maps fuck you... so pass the word...

oh yeah... wanna exchange links? :o)

cj@deluxeelectronics.com

google is slower

but just create a HTML simple sitemap

versus the xml they ask for or the .gz

then the bot grabs it no problem

Halfdeck 2005-08-29 10:48 AM
The reason why google doesn't like cj's page is because the html code has alot of errors, not because he submitted a site map.

Surfn 2005-08-29 12:30 PM
Quote:
Originally Posted by Halfdeck
The reason why google doesn't like cj's page is because the html code has alot of errors, not because he submitted a site map.
LOL true enough

troy 2005-08-31 03:41 PM
Quote:
i think google is going down hill and yahoo is going to fuck them up :o)
I hope so :D

SH 2005-08-31 11:45 PM
yahoo is only good for finding spam, think msn search will be the next google in few years when its build in new windows version :)

Surfn 2005-09-01 01:37 AM
If you are having a hard time with google or yahoo listing (getting your page results in the top 20) today you will have a hard time with any new SE that comes along. This isn't rocket science, close maybe. All the documentation is public on what each SE needs to list your sites.

It seems that people read less and less and whine more and more :(

sean 2005-09-02 04:39 AM
Quote:
Originally Posted by SH
yahoo is only good for finding spam, think msn search will be the next google in few years when its build in new windows version :)
True.I'm already getting more and more hits from MSN.


All times are GMT -4. The time now is 11:16 PM.

Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
© Greenguy Marketing Inc