Information Overload Revisited
Jim Harris in Sponsored Blog Posts tagged Best of 2011, Big Data, Enterprise CIO Forum, HP Thursday, December 15, 2011 at 11:00AM This blog post is sponsored by the Enterprise CIO Forum and HP.
Information Overload is a term invoked regularly during discussions about the data deluge of the Information Age, which has created a 24 hours a day, 7 days a week, 365 days a year, world-wide whirlwind of constant information flow, where the very air we breath is literally teeming with digital data streams — continually inundating us with new, and new types of, information.
Information overload generally refers to how too much information can overwhelm our ability to understand an issue, and can even disable our decision making in regards to that issue (this latter aspect is generally referred to as Analysis Paralysis).
But we often forget that the term is over 40 years old. It was popularized by Alvin Toffler in his bestselling book Future Shock, which was published in 1970, back when the Internet was still in its infancy, and long before the Internet’s progeny would give birth to the clouds contributing to the present, potentially perpetual, forecast for data precipitation.
A related term that has become big in the data management industry is Big Data, which, as Gartner Research explains, although the term acknowledges the exponential growth, availability, and use of information in today’s data-rich landscape, big data is about more than just data volume. Data variety (i.e., structured, semi-structured, and unstructured data, as well as other types, such as the sensor data emanating from the Internet of Things) and data velocity (i.e., how fast data is being produced and how fast the data must be processed to meet demand) are also key characteristics of the big challenges of big data.
John Dodge and Bob Gourley recently discussed big data on Enterprise CIO Forum Radio, where Gourley explained that big data is essentially “the data that your enterprise is not currently able to do analysis over.” This point resonates with a similar one made by Bill Laberis, who recently discussed new global research where half of the companies polled responded that they cannot effectively deal with analyzing the rising tide of data available to them.
Most of the big angst about big data comes from this fear that organizations are not tapping the potential business value of all that data not currently being included in their analytics and decision making. This reminds me of psychologist Herbert Simon, who won the 1978 Nobel Prize in Economics for his pioneering research on decision making, which included comparing and contrasting the decision-making strategies of maximizing and satisficing (a term that combines satisfying with sufficing).
Simon explained that a maximizer is like a perfectionist who considers all the data they can find because they need to be assured that their decision was the best that could be made. This creates a psychologically daunting task, especially as the amount of available data constantly increases (again, note that this observation was made over 40 years ago). The alternative is to be a satisficer, someone who attempts to meet criteria for adequacy rather than identify an optimal solution. And especially when time is a critical factor, such as it is with the real-time decision making demanded by a constantly changing business world.
Big data strategies will also have to compare and contrast maximizing and satisficing. Maximizers, if driven by their angst about all that data they are not analyzing, might succumb to information overload. Satisficers, if driven by information optimization, might sufficiently integrate just enough of big data into their business analytics in a way that satisfies specific business needs.
As big data forces us to revisit information overload, it may be useful for us to remember that originally the primary concern was not about the increasing amount of information, but instead the increasing access to information. As Clay Shirky succinctly stated, “It’s not information overload, it’s filter failure.” So, to harness the business value of big data, we will need better filters, which may ultimately make for the entire distinction between information overload and information optimization.
This blog post is sponsored by the Enterprise CIO Forum and HP.
Related Posts
The Good, the Bad, and the Secure
Securing your Digital Fortress
Shadow IT and the New Prometheus
Are Cloud Providers the Bounty Hunters of IT?
The Diderot Effect of New Technology
The IT Consumerization Conundrum
The IT Prime Directive of Business First Contact
A Sadie Hawkins Dance of Business Transformation
Are Applications the La Brea Tar Pits for Data?
Why does the sun never set on legacy applications?
The IT Pendulum and the Federated Future of IT
Reader Comments (1)
From the LinkedIn Group for Enterprise CIO Forum, Pearl Zhu commented:
“As usual, another insightful blog to articulate Big Data analytics elegantly, resourcefully & philosophically, I like satisficing, a hybrid word to well describe the big data journey: unleash big data's potential, also avoid the possible big distraction.
The traditional business intelligence (BI) based on limited, structured data may deliver the perfect chart more practically, for big data, perfection may not be necessary. The insight, the wisdom, and the trend just need to be sufficient enough to map with the management brain for the forward-looking organization.”
And Paul Calento commented:
“Totally agree about the information overload issue. An example of the daunting nature of the Big Data challenge is that many CIOs are just beginning to investigate this trend.
Key issue with Big Data is relevance — dealing with structured and unstructured data that wasn't necessarily designed to be used for the kinds of predictive decision making it now is.
Much like Siri benefits from use from millions of users, the same can be (hopefully) said for Big Data. Use it and the right filters will be developed and refined, based on volume of responses.”
And I responded:
The crowd-sourced filtering of Big Data is an excellent point.
It reminds me of CAPTCHA, which has been used to digitize old print books while also reducing spam. However, the success of CAPTCHA is also, in a way, and somewhat ironically, helping spam-bot algorithms get better at recognizing the words in CAPTCHA, and therefore pass the spam filter. In other words, our spam filters are helping to evolve more advanced spam.
Hopefully, our attempts to filter Big Data for positive uses doesn't have any similar untended negative consequences :-)