Information overload
My friends look at me like I'm crazy when I tell them that I'm trying to find an RSS/Atom feed aggregator that won't choke when I try to subscribe to over 350 feeds. I wonder what they will think when they find out that I'm also trying to keep track of:
- 466 Twitter feeds,
- 188 Facebook friends,
- 30 podcasts,
- 10 IRC channels,
- 6 Meetup groups,
- 3 instant messaging networks,
- 2 voicemail boxes, and
- 1 email address that sorts 2000 messages per day through 231 procmail rules and a spam filter.
Does that sound like information overload?
Let's put it into perspective: The ATLAS detector (part of the Large Hadron Collider) generates 25 megabytes of output every time a packet of protons crosses it, which happens 40 million times per second. That's a whopping 1,000,000 GB every second!
Today's technology can't store that much data for very long. Even assuming the availability of $100 1TB hard disk drives that could store the information fast enough (they can't) and that were 100% reliable (they aren't), ATLAS would need 1.4 million of these drives (and a budget of $144 million) per day just for this initial data storage.
That is information overload!
The scientists and engineers working on ATLAS are well aware of these physical limitations, and they deal with them by using a clever multi-stage process that ultimately figures out what's important and discards the remaining 99.99997% of the data, resulting in a data rate of 'only' 320 MB per second (see page 5 of the linked PDF).
Think about those numbers. That is the state of the art.
Yet, I struggle with a handful of RSS feeds and a couple of social networking sites! I've thought about just pruning my subscriptions and changing my email address every few years like everyone else does, but I'm too embarrassed to admit to myself that I am so far behind the state of the art.
Darnit, I'm a programmer! If the ATLAS scientists can do what they did, I should easily be able to put together something that can sift through a measly fraction of the data while throwing away much less of it, especially when I can let it work for seconds or even minutes instead of nanoseconds. It's a simple, routine automation task.
Right?
Recent Posts
- When you’re making a list of what’s naughty and nice, check it twice.
- Atheist Week on Facebook – from an apartment in Ottawa
- Information overload
- “Feeling blue? Look to hugs before religion” – G&M
- PyCrypto 2.1.0 released
Blogroll
- Academic Hacker News
- Accidentally in Code
- Excess Copyright
- Greta Christina's Blog
- Groklaw
- Linux Hater's Blog
- Michael Geist's Blog
- OkTrends
- Open Access News
- Ottawa Skeptics
- Pharyngula
- Wikileaks
Recent Comments
- Dwayne Litzenberger on When you’re making a list of what’s naughty and nice, check it twice.
- WTL on When you’re making a list of what’s naughty and nice, check it twice.
- Jevin on Information overload
- Dwayne Litzenberger on Information overload
- Jevin on Information overload
Archives
December 26th, 2009 - 09:19
If you manage to do this… I’d use it too :-)
February 6th, 2010 - 17:19
Dude! I remember we were talking about this at OGRE December! I discovered “Feed a Fever” for my RSS reading and it really helps. It basically watches the links in each post and will prioritize your list based on how many times a link was quoted:
http://feedafever.com/
Let me know what you think.
February 6th, 2010 - 18:50
Hi Jevin!
I bought and tried FeedAFever a few months ago, but I stopped using it after a couple of days; it’s just not what I’m looking for. Digg, Slashdot, Facebook, and Twitter are already good at pointing me to things that are popular, so FeedAFever doesn’t really add much, and it adds the cost of being yet another site I would need to visit on a regular basis.
I’ve given up on finding a single program that’s going to solve my information management problem. What I’m looking for now are tools that will help me to construct arbitrary criteria for sifting through information from arbitrary sources and to present the results in arbitrary ways. These really should be separate tools with simple, well-defined APIs.
Niche applications like FeedAFever and FriendFeed are designed to interact directly with end-users on a regular basis, but my fundamental problem as an end-user is that I already have too many things I need to interact with!
Other people have suggested web-based services (e.g. Google Reader) but that’s not a direction I want to go. Some of the information I need to sift through is private (like my email, telephone, income/expenses, etc.). Also, the median lifespan of these services is so short that I’m wary of relying on them for something so critical to my day-to-day life.
I’m really looking for something with the flexibility of the Unix command-line, rather than another fancy webapp, unless that webapp is primarily concerned with presenting information generated from other sources.
Thanks for commenting!
February 6th, 2010 - 20:19
I hear you. I think if someone did come up with a clean console solution, they would only have two customers: you.. and me.
That said, I’ve been reading lots of Tim Ferris’ stuff. It totally goes against information overload. The dude doesn’t even read the newspaper! Let I have already have more and more data to go through.
If we had a solution.. world domination would be in reach for sure.