Lets Talk Aggregator

[Useless]

From what I understand, the ever-decreasing speed of an aggregator as it ages is due to the fact that many of them (aggregation scripts) go through and scan what you have stored in your database from feed A and compare it that to feed A's present state. That's a lot of work, especially when you have that many feeds being handled and a now monstrous database. A more efficient aggregator would look at the post dates and add only what's new from a feed, instead of checking to see if old posts have been updated/edited. If they didn't get it right the first time, fuck 'em.

[NobleSavage]

I'd like one coded in C that generates new static pages every 2-5 minutes.

[cash29]

Originally Posted by NobleSavage

I'd like one coded in C that generates new static pages every 2-5 minutes.

There is a python-based script that does just what you're asking for (boviously, not in C):
http://www.planetplanet.org/

Bu if you know some C, you can always optimize the parts that need to be

[NobleSavage]

Originally Posted by cash29

There is a python-based script that does just what you're asking for (boviously, not in C):
http://www.planetplanet.org/

Bu if you know some C, you can always optimize the parts that need to be

Hey Cash,

That looks like a nice script. I'll have to give it a shot. And python is just cool. Thanks!

[cash29]

Originally Posted by Useless Warrior

From what I understand, the ever-decreasing speed of an aggregator as it ages is due to the fact that many of them (aggregation scripts) go through and scan what you have stored in your database from feed A and compare it that to feed A's present state. That's a lot of work, especially when you have that many feeds being handled and a now monstrous database. A more efficient aggregator would look at the post dates and add only what's new from a feed, instead of checking to see if old posts have been updated/edited. If they didn't get it right the first time, fuck 'em.

No, you CANNOT look at the post dates to determine uniqueness of posts. All I would have to do to spam your aggregator would be to update the date of my post without changing its content.

Most feed aggregators should check for the server's Last-Modified/If-Modified-Since response or even better, its ETag/If-None-Match response. These responses tell you if a feed item has changed since the aggregator last checked it. You can read a little more about these here:
http://diveintopython.org/http_web_s..._features.html

[NobleSavage]

Originally Posted by cash29

Most feed aggregators should check for the server's Last-Modified/If-Modified-Since response or even better, its ETag/If-None-Match response. These responses tell you if a feed item has changed since the aggregator last checked it. You can read a little more about these here:
http://diveintopython.org/http_web_s..._features.html

Couldn't a smart spamer alter his server response codes?

[cash29]

Originally Posted by NobleSavage

Couldn't a smart spamer alter his server response codes?

If he can hack on Apache's source code, then yes. Although, this seems somewhat unlikey: how many webmasters have access to their own Apache? How many know enough C to make Apache behave in that way?

[NobleSavage]

Originally Posted by cash29

If he can hack on Apache's source code, then yes. Although, this seems somewhat unlikey: how many webmasters have access to their own Apache? How many know enough C to make Apache behave in that way?

Couldn't you just use:

<?php
header('Last-Modified: ' . $date_string ');
?>

[cash29]

Originally Posted by NobleSavage

Couldn't you just use:

<?php
header('Last-Modified: ' . $date_string ');
?>

PHP can do that ? Oops, my bad, just goes to show how much of it I really know

[Useless]

Originally Posted by cash29

No, you CANNOT look at the post dates to determine uniqueness of posts. All I would have to do to spam your aggregator would be to update the date of my post without changing its content.

Then you drop their feed. Done! One can't expect to let a site to run itself and end up with quality. And if one forces an aggregator to look at every post in a heavy database, you must except that the slow down is going to occur.

[cash29]

Originally Posted by Useless Warrior

Then you drop their feed. Done! One can't expect to let a site to run itself and end up with quality. And if one forces an aggregator to look at every post in a heavy database, you must except that the slow down is going to occur.

Good point. The best solution to spam isn't a technical one, just don't allow spammy blogs on your aggregator
After this discussion, I'm thinking:
* it makes more and more sense to just keep at 50 items or so per feed at any given time. I think this should be enough to keep visitors busy and should keep your script running not too badly.

* Writing your own aggregator is the way to go. Most of the ones I've used were too immature and didn't scale well. Gregarius and Lilina (PHP-based) would both start to choke at around 15,000 feed items (I read a lot of tech blogs).