Data-Driven Intuition

The book The House Advantage: Playing the Odds to Win Big In Business by Jeffrey Ma should be required reading for anyone working in the data management and business intelligence fields where we often oversimplify the business decision-making process by saying it’s either data-driven or intuition-driven—and strongly emphasizing that using data is always better than using intuition. Although Ma is definitely an advocate for data-driven decision making, toward the end of his book he also acknowledges that there are times when somewhat of a middle ground between data and intuition is called for.

“In the real world,” Ma explained, “the number of variables are often too numerous and the sample size too small to create perfect models. But that doesn’t mean you should ignore the data. It just means that you always strive to find the best data—the most relevant data with which to make the best decisions. I call this data-driven intuition, and it is a process that many successful people implement when faced with a situation where using a purely data-driven statistical model is impossible.”

The definition of intuition is “a direct perception of truth or fact independent of any reasoning process,” i.e., what is often referred to as “going with your gut.” In his quest to find a successful individual who truly used their gut and nothing else to make decisions, Ma discovered he needed to redefine intuition as “a direct perception of truth or fact independent of any documented reasoning process” because “no one successful truly makes decisions without some reasoning process. They may not want to spend the time to explain that reasoning process, or they may not have the sufficient information to document that reasoning process, but it’s clear in all cases I explored that there was a method behind the madness.”

When business leaders talk about relying on their intuition, which is based on their considerable personal experience and professional expertise that has guided their business success to date, most data management and business intelligence professionals lament that the organization’s leadership doesn’t embrace a culture of data-driven decision making.

However, not all data is stored in a database, or a NoSQL data store. In fact, some of the world’s most valuable (and most unusually structured) data is stored within the human mind.

“Some of the world’s most valuable data is stored within the human mind.”

As Ma explained, “every good decision has some data behind it, as well as a thorough examination of the specific case at hand. It might not be data that sits in a spreadsheet or an analysis performed by a computer, but it is science much more than art. Helping people understand what they are doing when they make what they think are intuitive decisions will help them appreciate the true value of using analytics.”

Perhaps we have always been data scientists. Perhaps our intuition has always been more data-driven than we gave it credit for.

“Analytics has always been about making better decisions—any way that we can.”

Maybe data-driven intuition can bridge the data-intuition divide in analytics. Or maybe it can remind us that analytics has never been about data or intuition. Analytics has always been about making better decisions—any way that we can.