Dare posted up a comment on one of my posts about RSSBandit and the problems I was having. He asked what I would suggest to fix the problem.
Before I make my suggestions I want to point out that RSSBandit is free software and Dare devotes serious time to maintaining the software so its not fair for me to criticise his work. The code itself is awesome, it just has a problem when it reaches the number of feeds that I was trying to aggregate.
That said, I do have a few suggestions. The first is to make UI updates high value, there are lots of locations in the code where a call is marshalled across to the UI just to update a single statistic. This is fine for your typical feed load but when you have a thousand feeds the system spends a lot of time synchronising threads for you.
I would suggested having some kind of statistics system where periodically (on idle) the treeview nodes are updated all in one go. To compliment this it may be more efficient to store pre-computed statistics along with the applications data files so that when the program is loaded it shows what it last knew the statistics to be.
The background thread would then just walk the feeds that have been updated since the last UI refresh and update that statistics file. Diagramatically it would look something like this:
By decoupling the UI update from the statistics refresh you give yourself the ability to optimise each algorithm seperately. For example, when refreshing feeds (not necessarily done of this background thread – I know) it may be more efficient to group hosts together.
I subscribe to quite a few blogs at http://blogs.msdn.com but like a lot of hosts it limits you to 2–4 simultaneous connections as defined in one of the HTTP specs. So if you do multithreaded fetches of RSS feeds you could essentially have a bunch of then in a wait state while they are hung up waiting for the web-services to hear their request.
The feasability of doing all of this may be reduced if Dare upgrades RSSBandit to work with Windows Vista because it can exploit the RSS platform that is part of Internet Explorer and it might be more appropriate for Dare to make us greedy feeders wait until then.