The Dichotomy Paradox, Data Quality and Zero Defects
/As Joseph Mazur explains in Zeno’s Paradox, the ancient Greek philosopher Zeno constructed a series of logical paradoxes to prove that motion is impossible, which today remain on the cutting edge of our investigations into the fabric of space and time.
One of the paradoxes is known as the Dichotomy:
“A moving object will never reach any given point, because however near it may be, it must always first accomplish a halfway stage, and then the halfway stage of what is left and so on, and this series has no end. Therefore, the object can never reach the end of any given distance.”
Of course, this paradox sounds silly. After all, reaching a given point like the finish line in a race is reachable in real life since people win races all the time. However, in theory, the mathematics is maddeningly sound, since it creates an infinite series of steps between the starting point and the finish line—and an infinite number of steps creates a journey that can never end.
Furthermore, this theoretical race cannot even begin, since in order to reach the first step, the recursive nature of this paradox proves that we would never reach the point of completing the first step. Hence, the paradoxical conclusion is any travel over any finite distance can neither be completed nor begun, and so all motion must be an illusion. Some of the greatest minds in history (from Galileo to Einstein to Stephen Hawking) have tackled the Dichotomy Paradox—but without being able to disprove it.
Data Quality and Zero Defects
The given point that many enterprise initiatives attempt to reach with data quality is 100% with a metric such as data accuracy. Leaving aside (in this post) the fact that any data quality metric without a tangible business context provides no business value, 100% data quality (aka Zero Defects) is an unreachable destination—no matter how close you get or how long you try to reach it.
Zero Defects is a laudable goal—but its theory and practice comes from manufacturing quality. However, I have always been of the opinion, unpopular among some of my peers, that manufacturing quality and data quality are very different disciplines, and although there is much to be learned from studying the theories of manufacturing quality, I believe that brute forcing those theories onto data quality is impractical and fundamentally flawed (and I’ve even said so in verse: To Our Data Perfectionists).
The given point that enterprise initiatives should actually be attempting to reach is data-driven solutions for business problems.
Advocates of Zero Defects argue that, in theory, defect-free data should be fit to serve as the basis for every possible business use, enabling a data-driven solution for any business problem. However, in practice, business uses for data, as well as business itself, is always evolving. Therefore, business problems are dynamic problems that do not have—nor do they require—perfect solutions.
Although the Dichotomy Paradox proves motion is theoretically impossible, our physical motion practically proves otherwise. Has your data quality practice become motionless by trying to prove that Zero Defects is more than just theoretically possible?
