Lets Talk Aggregator

[cash29]

Originally Posted by Useless Warrior

Then you drop their feed. Done! One can't expect to let a site to run itself and end up with quality. And if one forces an aggregator to look at every post in a heavy database, you must except that the slow down is going to occur.

Good point. The best solution to spam isn't a technical one, just don't allow spammy blogs on your aggregator
After this discussion, I'm thinking:
* it makes more and more sense to just keep at 50 items or so per feed at any given time. I think this should be enough to keep visitors busy and should keep your script running not too badly.

* Writing your own aggregator is the way to go. Most of the ones I've used were too immature and didn't scale well. Gregarius and Lilina (PHP-based) would both start to choke at around 15,000 feed items (I read a lot of tech blogs).