Magic Elephants, Data Psychics, and Invisible Gorillas
/This blog post is sponsored by the Enterprise CIO Forum and HP.
A recent Forbes article predicts Big Data will be a $50 billion market by 2017, and Michael Friedenberg recently blogged how the rise of big data is generating buzz about Hadoop (which I call the Magic Elephant): “It certainly looks like the Holy Grail for organizing unstructured data, so it’s no wonder everyone is jumping on this bandwagon. So get ready for Hadoopalooza 2012.”
John Burke recently blogged about the role of big data helping CIOs “figure out how to handle the new, the unusual, and the unexpected as an opportunity to focus more clearly on how to bring new levels of order to their traditional structured data.”
As I have previously blogged, many big data proponents (especially the Big Data Lebowski vendors selling Hadoop solutions) extol its virtues as if big data provides clairvoyant business insight, as if big data was the Data Psychic of the Information Age.
But a recent New York Times article opened with the story of a statistician working for a large retail chain being asked by his marketing colleagues: “If we wanted to figure out if a customer is pregnant, even if she didn’t want us to know, can you do that?” As Eric Siegel of Predictive Analytics World is quoted in the article, “we’re living through a golden age of behavioral research. It’s amazing how much we can figure out about how people think now.”
So, perhaps calling big data psychic is not so far-fetched after all. However, the potential of predictive analytics exemplifies why one of the biggest implications about big data is the data privacy concerns it raises.
Although it’s amazing (and scary) how much the Data Psychic can figure out about how we think (and work, shop, vote, love), it’s equally amazing (and scary) how much Psychology is figuring out about how we think, how we behave, and how we decide.
As I recently blogged about WYSIATI (“what you see is all there is” from Daniel Kahneman’s book Thinking, Fast and Slow), when you are using big data to make business decisions, what you are looking for can greatly influence what you are looking at (and vice versa). But this natural human tendency could cause you miss the Invisible Gorilla walking across your screen.
If you are unfamiliar with that psychology experiment, which was created by Christopher Chabris and Daniel Simons, authors of the book The Invisible Gorilla: How Our Intuitions Deceive Us, then I recommend going to theinvisiblegorilla.com/videos.html. (By the way, before I was familiar with its premise, the first time I watched the video, I did not see the guy in the gorilla suit, and now when I watch the video, seeing the “invisible gorilla” distracts me, causing me to not count the number of passes correctly.)
In his book Incognito: The Secret Lives of the Brain, David Eagleman explained how our brain samples just a small bit of the physical world, making time-saving assumptions and seeing only as well as it needs to. As our eyes interrogate the world, they optimize their strategy for the incoming data, arbitrating a battle between the conflicting information. What we see is not what is really out there, but instead only a moment-by-moment version of which perception is winning over the others. Our perception works not by building up bits of captured data, but instead by matching our expectations to the incoming sensory data.
I don’t doubt the Magic Elephants and Data Psychics provide the potential to envision and analyze almost anything happening within the complex and constantly changing business world — as well as the professional and personal lives of the people in it.
But I am concerned that information optimization driven by the biases of our human intuition and perception will only match our expectations to those fast-moving large volumes of various data, thereby causing us to not see many of the Invisible Gorillas.
Although this has always been a business intelligence concern, as technological advancements improve our data analytical tools, we must not lose sight of the fact that tools and data remain only as effective (and as beneficent) as the humans who wield them.
This blog post is sponsored by the Enterprise CIO Forum and HP.
Related Posts
Neither the I Nor the T is Magic
Information Overload Revisited
HoardaBytes and the Big Data Lebowski
A Decision Needle in a Data Haystack
Dot Collectors and Dot Connectors
