Quote:
|
I know why they are EX-engineers.. because they are crap!!!
|
One of them, Ann Patterson, is probably responsible for creating Google's supplemental results, which allows Google to crawl a ton of more pages than it used to. Using (internal) PageRank as a crawl-meter, Google tends to crawl lower PageRank pages less often than pages with higher PageRank. Pages in the supplemental index (invisible now, but still exists) are on a different crawl queue; Google used to crawl supp pages every 3-4 months. That delay allowed Google to cover more webspace without exhausting resources.
That algo is probably the cornerstone of Cuil.com, which prouds itself on being able to index more pages off the web than any other engine.