Yahoo vs. Google robots???

dunkin · 2006-02-11, 11:50 AM

Hey gang,
Can anyone explain why the Yahoo robot crawls less the Google (252 hits to 382 hits) but hits the robot text over 10 times to one than Google does ( 235 hits to 22 hits)

I posted a copy of the latest stats here. These numbers are for the month of February:
http://www.dunkinsworld.com/sestats.htm

Not to long ago, it was the other way around. I haven't changed a thing, other than some general tweaking and such.
Am I missing something or is it just a google thing?

Any suggestions?
Thanks again,
Dunkin

dunkin · 2006-02-14, 12:17 PM

anyone?

cd34 · 2006-02-14, 12:51 PM

I think google has bots which usually crawl a number of pages at a time -- whereas yahoo seems to crawl one or two pages at a time. As a result, if google and yahoo only grab a robots.txt per 'session', yahoo will pull robots.txt many more times because they are getting fewer pages at a time.

So, google may have hit you with only 22 different sessions that each grabbed 17 pages and yahoo may have hit you with 235 sessions that grabbed one or two pages at a time.

You might also look at the number of Unique IPs that google sent versus the Unique IPs that yahoo sent. Perhaps more machines from yahoo hit your site.

As for the number of pages, who knows how those engines decide what to crawl. Google may have already cached your page structure and Yahoo may be still discovering pieces of it. Hard to say really.

All conjecture and guesses.

dunkin · 2006-02-14, 01:43 PM

Hey cd,
Thanks for the info. I think you are correct that Google crawls more pages in one session as compared to yahoo. That would explain Google's new site map procedures (listing all pages you want to be crawled in a text file).

I wanted to make sure my Google site map info/files were correct.

Also, that my meta tags were correct and properly used.

Freaking serch engines

Thanks again,
Dunkin

2006-02-11, 11:50 AM	#1
dunkin Just because I don't care doesn't mean I don't understand! Join Date: Feb 2004 Posts: 92	Yahoo vs. Google robots??? Hey gang, Can anyone explain why the Yahoo robot crawls less the Google (252 hits to 382 hits) but hits the robot text over 10 times to one than Google does ( 235 hits to 22 hits) I posted a copy of the latest stats here. These numbers are for the month of February: http://www.dunkinsworld.com/sestats.htm Not to long ago, it was the other way around. I haven't changed a thing, other than some general tweaking and such. Am I missing something or is it just a google thing? Any suggestions? Thanks again, Dunkin __________________ Dunkins World - Accepting gallery submits & link trades.

2006-02-14, 12:17 PM	#2
dunkin Just because I don't care doesn't mean I don't understand! Join Date: Feb 2004 Posts: 92	anyone? __________________ Dunkins World - Accepting gallery submits & link trades.

2006-02-14, 12:51 PM	#3
cd34 a.k.a. Sparky Join Date: Sep 2004 Location: West Palm Beach, FL, USA Posts: 2,396	I think google has bots which usually crawl a number of pages at a time -- whereas yahoo seems to crawl one or two pages at a time. As a result, if google and yahoo only grab a robots.txt per 'session', yahoo will pull robots.txt many more times because they are getting fewer pages at a time. So, google may have hit you with only 22 different sessions that each grabbed 17 pages and yahoo may have hit you with 235 sessions that grabbed one or two pages at a time. You might also look at the number of Unique IPs that google sent versus the Unique IPs that yahoo sent. Perhaps more machines from yahoo hit your site. As for the number of pages, who knows how those engines decide what to crawl. Google may have already cached your page structure and Yahoo may be still discovering pieces of it. Hard to say really. All conjecture and guesses. __________________ SnapReplay.com a different way to share photos - iPhone & Android

2006-02-14, 01:43 PM	#4
dunkin Just because I don't care doesn't mean I don't understand! Join Date: Feb 2004 Posts: 92	Hey cd, Thanks for the info. I think you are correct that Google crawls more pages in one session as compared to yahoo. That would explain Google's new site map procedures (listing all pages you want to be crawled in a text file). I wanted to make sure my Google site map info/files were correct. Also, that my meta tags were correct and properly used. Freaking serch engines Thanks again, Dunkin __________________ Dunkins World - Accepting gallery submits & link trades.