Magic Elephants, Data Psychics, and Invisible Gorillas
Jim Harris in Books, Sponsored Blog Posts tagged Best of 2012, Big Data, Business Intelligence, Data Privacy, Enterprise CIO Forum, HP, Predictive Analytics Tuesday, February 21, 2012 at 3:00AM This blog post is sponsored by the Enterprise CIO Forum and HP.
A recent Forbes article predicts Big Data will be a $50 billion market by 2017, and Michael Friedenberg recently blogged how the rise of big data is generating buzz about Hadoop (which I call the Magic Elephant): “It certainly looks like the Holy Grail for organizing unstructured data, so it’s no wonder everyone is jumping on this bandwagon. So get ready for Hadoopalooza 2012.”
John Burke recently blogged about the role of big data helping CIOs “figure out how to handle the new, the unusual, and the unexpected as an opportunity to focus more clearly on how to bring new levels of order to their traditional structured data.”
As I have previously blogged, many big data proponents (especially the Big Data Lebowski vendors selling Hadoop solutions) extol its virtues as if big data provides clairvoyant business insight, as if big data was the Data Psychic of the Information Age.
But a recent New York Times article opened with the story of a statistician working for a large retail chain being asked by his marketing colleagues: “If we wanted to figure out if a customer is pregnant, even if she didn’t want us to know, can you do that?” As Eric Siegel of Predictive Analytics World is quoted in the article, “we’re living through a golden age of behavioral research. It’s amazing how much we can figure out about how people think now.”
So, perhaps calling big data psychic is not so far-fetched after all. However, the potential of predictive analytics exemplifies why one of the biggest implications about big data is the data privacy concerns it raises.
Although it’s amazing (and scary) how much the Data Psychic can figure out about how we think (and work, shop, vote, love), it’s equally amazing (and scary) how much Psychology is figuring out about how we think, how we behave, and how we decide.
As I recently blogged about WYSIATI (“what you see is all there is” from Daniel Kahneman’s book Thinking, Fast and Slow), when you are using big data to make business decisions, what you are looking for can greatly influence what you are looking at (and vice versa). But this natural human tendency could cause you miss the Invisible Gorilla walking across your screen.
If you are unfamiliar with that psychology experiment, which was created by Christopher Chabris and Daniel Simons, authors of the book The Invisible Gorilla: How Our Intuitions Deceive Us, then I recommend going to theinvisiblegorilla.com/videos.html. (By the way, before I was familiar with its premise, the first time I watched the video, I did not see the guy in the gorilla suit, and now when I watch the video, seeing the “invisible gorilla” distracts me, causing me to not count the number of passes correctly.)
In his book Incognito: The Secret Lives of the Brain, David Eagleman explained how our brain samples just a small bit of the physical world, making time-saving assumptions and seeing only as well as it needs to. As our eyes interrogate the world, they optimize their strategy for the incoming data, arbitrating a battle between the conflicting information. What we see is not what is really out there, but instead only a moment-by-moment version of which perception is winning over the others. Our perception works not by building up bits of captured data, but instead by matching our expectations to the incoming sensory data.
I don’t doubt the Magic Elephants and Data Psychics provide the potential to envision and analyze almost anything happening within the complex and constantly changing business world — as well as the professional and personal lives of the people in it.
But I am concerned that information optimization driven by the biases of our human intuition and perception will only match our expectations to those fast-moving large volumes of various data, thereby causing us to not see many of the Invisible Gorillas.
Although this has always been a business intelligence concern, as technological advancements improve our data analytical tools, we must not lose sight of the fact that tools and data remain only as effective (and as beneficent) as the humans who wield them.
This blog post is sponsored by the Enterprise CIO Forum and HP.
Related Posts
Neither the I Nor the T is Magic
Information Overload Revisited
HoardaBytes and the Big Data Lebowski
A Decision Needle in a Data Haystack
Dot Collectors and Dot Connectors
Reader Comments (3)
Insightful post as usual Jim.
Loved the phrase from David Eagleman, "What we see is not what is really out there, but instead only a moment-by-moment version of which perception is winning over the others."
Reminds me of the old joke "They are not sure who discovered water, but are pretty sure it wasn't fish".
It seems that nearly everybody working with information is inured to the poor data quality, and it no longer even registers. Thus the need to share, tweet and re-post your blog articles to raise people's consciousness of the disabling data quality surrounding all of us. :)
Cheers, Gordon
From the LinkedIn Group for Enterprise CIO Forum, Pearl Zhu commented:
“Hi, Jim, as usual, your world of data is so enriched and vivid, from data poem to data symphony, from magic elephants to data psychics, and may just further verify that we may live in the era of the Data Renaissance.
As I blogged in my post Three Big WHYs for Big Data, the biggest Big Data challenge facing business today could be the distraction it may bring up, the full life cycle data management, the talent to master the data, and how to take advantage of resources wisely to capture the business insight and foresight that organizations desperately need these days.”
And I responded:
Thanks for your great comment, Pearl.
And thanks for the link to your excellent blog post about big data!
Best Regards,
Jim
You and I have a similar reading list, I've read all of those articles recently, including the psychology focused books! I enjoy your creative and entertaining labeling of aspects that we've always focused on but are now being so hyped up these days.
Invisible Gorillas and other distractions need this cool mind and OCDQ perspectives. Thanks!