Big Data Quality, Then and Now

Other Ways to Listen: bit.ly/listen-dbp

A decade ago, just before the beginning of the data science hype cycle was the big data hype cycle. At that time I had the privilege of sitting down with Ph.D. Statistician Dr. Thomas C. Redman (aka the “Data Doc”).

We discussed whether data quality matters less in larger data sets, if statistical outliers represent business insights or data quality issues, statistical sampling errors versus measurement calibration errors, mistaking signal for noise (i.e., good data for bad data), and whether or not the principles and practices of true “data scientists” will truly be embraced by an organization’s business leaders.

This episode is an edited and slightly shortened version of that discussion, which even though it is from ten years ago, I think it still provides good insight into big data quality, then and now.

Extended Show Note: One example of a measurement calibration error that was mentioned during this discussion is the faster-than-light neutrino anomaly, which you can read more about on Wikipedia.