Quote:
Originally Posted by Useless Warrior
From what I understand, the ever-decreasing speed of an aggregator as it ages is due to the fact that many of them (aggregation scripts) go through and scan what you have stored in your database from feed A and compare it that to feed A's present state. That's a lot of work, especially when you have that many feeds being handled and a now monstrous database. A more efficient aggregator would look at the post dates and add only what's new from a feed, instead of checking to see if old posts have been updated/edited. If they didn't get it right the first time, fuck 'em.
|
No, you CANNOT look at the post dates to determine uniqueness of posts. All I would have to do to spam your aggregator would be to update the date of my post without changing its content.
Most feed aggregators should check for the server's Last-Modified/If-Modified-Since response or even better, its ETag/If-None-Match response. These responses tell you if a feed item has changed since the aggregator last checked it. You can read a little more about these here:
http://diveintopython.org/http_web_s..._features.html