Lets Talk Aggregator

[Useless]

Originally Posted by cash29

No, you CANNOT look at the post dates to determine uniqueness of posts. All I would have to do to spam your aggregator would be to update the date of my post without changing its content.

Then you drop their feed. Done! One can't expect to let a site to run itself and end up with quality. And if one forces an aggregator to look at every post in a heavy database, you must except that the slow down is going to occur.

[cash29]

Originally Posted by Useless Warrior

Then you drop their feed. Done! One can't expect to let a site to run itself and end up with quality. And if one forces an aggregator to look at every post in a heavy database, you must except that the slow down is going to occur.

Good point. The best solution to spam isn't a technical one, just don't allow spammy blogs on your aggregator
After this discussion, I'm thinking:
* it makes more and more sense to just keep at 50 items or so per feed at any given time. I think this should be enough to keep visitors busy and should keep your script running not too badly.

* Writing your own aggregator is the way to go. Most of the ones I've used were too immature and didn't scale well. Gregarius and Lilina (PHP-based) would both start to choke at around 15,000 feed items (I read a lot of tech blogs).