Fear has Replaced Apathy as the Number One Enemy of Data
/Background: “Apathy is the enemy of data quality”
I began work on data quality in the late 1980s at the great Bell Laboratories. We worked in partnership with a couple of AT&T groups and made rapid strides. The AT&T groups that applied our methods made order-of-magnitude improvements and, in so doing, saved tens of millions of dollars per year. Even better, senior leadership found the improved decision-making that resulted was almost certainly worth more than that, though we never could figure out how to quantify these benefits.
Since 1996, I have advised leading companies in telecom, finance, oil and gas, consumer products, and health care. The methods we developed at Bell Labs held up unreasonably well—those that applied them also made huge improvements and reaped great business results.
I thought our ideas and methods would spread like wildfire. Not so. Even when Department A had achieved a great success, neither Department B, right next door, nor Department A’s more senior management, showed much interest. Curiously perhaps, there was little opposition. Indeed, I can’t recall a single person who claimed high-quality data wasn’t important. They just did not make any efforts to improve. This led me to conclude, by about 2000, that apathy was the number one enemy of data quality.
An Ongoing Market Scan
Every few years I conduct a scan of the penetration of data quality across the broad spectrum of industry, government, and people’s psyches. As my interests expanded beyond quality, I expanded the scope of my scan to all things data. My goal is to understand progress, discontinuities in the forces that drive, and hold back, the penetration of data in industry. I especially wanted to identify industries that were ripe for data quality. As Blan Godfrey observed, the so-called quality revolution was better described as a series of industry-specific mini-revolutions, in consumer electronics, automobiles, and so forth, each in response to market demands.
While a lot had changed since 2000, even as late at 2010, I concluded that most everyone was apathetic about data.
Fear
To conduct my most recent scan, starting in 4Q 2014, I talked to dozens of people, caught up on the research reports (e.g., “The New Heroes of Big Data and Analytics”); re-read hundreds of stories appearing in the popular press (e.g., “The Data D. A.”), at least scanned the (many) new books on big data, data governance, and analytics (e.g., The Second Machine Age, Data-ism, and Big Data at Work); re-examined what I’d observed in teaching and consulting engagements across the US, Europe, and Asia, and attempted to put things in historical context.
The notion that “fear” was an important barrier to data grew in my mind slowly. It took root as I pondered the contrast between comments such as:
“We tried a half-dozen analytics projects and didn’t get anywhere. It’s not for us.”
“No one could attack us like Uber is attacking the taxi business.”
“We’re already data-driven. What’s the big deal?”
with
“It’s time to think about how data can improve our products.”
“If we could combine the hard data with the instincts of our veterans, we’d be world-beaters.”
“So far, we’ve only addressed data quality when we were in crisis. It’s time to get in front.”
The latter group was growing excited about the possibilities; the former group scared! Even better, the latter group no longer consisted solely of lovers of data such as myself, but regular people. Perhaps their excitement is exacerbating the levels of fear in others!
I formulated the headline result (fear has replaced apathy as the number one enemy of all things data) as a sort of trial balloon, and tried it out on several friendlies (people whom I trust, who will tell me when I’m wrong). Almost all concurred (note: A couple think the growth in the number of regular people excited about data should be the headline result). Still, without some understanding of causal factors, I was reluctant to go public.
Then it occurred to me that for many, perhaps most, fear was the rational response to what they’re seeing and hearing over the past few years. They can’t open their email or read the newspaper without seeing “big data this,” “predictive analytics that,” and “data-driven the other.” Unless they have pretty solid quantitative skills or are acquiring them rapidly, they should be scared!
Finally, I introduced my conclusion about fear during a panel session at a professional conference and received good feedback. Several told me that this conclusion gave them a new insight into behavior at their own companies, and many people urged me to publish the result (Anne Marie Smith was the most aggressive).
To be clear, there’s still plenty of apathy—people who think this data thing will blow over (as dozens of management fads have). I daresay the apathetics outnumber the fearfuls, though I suspect dealing with fear will be more difficult that dealing with apathy.
The following table provides my best qualitative summary as of June 30, 2015.
| 2010 | 2015 | |
|---|---|---|
| Fearful | Few | Many and growing |
| Apathetic | Vast majority | Many, but fewer |
| Truly Excited | A few, mostly in the data space | Still relatively few, but growing, and not just those in the data space |
Full disclosure: I make no pretense that my market scan can pass scientific scrutiny. I’ve made no attempts to talk to a random sample of people; many of the so-called research reports have serious flaws; I have no idea whether my literature review is complete; and I’m anything but an unbiased observer. And my scan does not have the precision to produce hard numbers. Finally, I expect considerable company-to-company, department-to-department, and work team-by work team variation. Those who prefer to discard these results can find ample grounds for doing so.
At the same time, my scan did not aim to pass scientific scrutiny or even serve as careful market research. It aims to identify sea changes, to see around corners, to gain insights into what’s next, and in this respect, it has more than met its objectives. And fear is too powerful an enemy to be ignored.
One final thought: Has data reached “the tipping point?”
Malcolm Gladwell introduced the term the tipping point to mark the time when a new idea or technology gains enough momentum that its deep penetration is assured. I’ve given careful thought to whether data has reached that point. Perhaps, the elevated levels of fear reflects a “sixth sense” that this is the case.
Maybe. But, I’m not ready to raise that flag. While the possibilities intoxicate, there are still too few concept-to-cash results. It’s going to take at least a couple more years of near-heroic effort, by courageous men and women to provide those results and reach the tipping point.
I’ll publish two companion pieces in the next several days that explore the steps business managers and lovers of data should take now. Look for them at hbr.org and dataversity.net.
Thomas C. Redman, Ph.D., “the Data Doc” and President of Navesink Consulting Group, advises organizations on their data and data quality programs. Redman is the author of Data Driven: Profiting from Your Most Important Business Asset, published by Harvard Business Press in 2008 and one of Library Journal’s best business books that year.
