In devising entire data-centric systems such Data Philosophers would be tasked with devising the constructs on a logical basis with the type of depth and variability to both capture the "real world" and humanistic aspects, but also allow such systems to attain a malleable quality that can be lacking when dealing with (especially customizable) constructions. Simple surveys and questions could be cross deployed, while the insight which are gleaned can become a cross-disciplinary affair -- a vexing problem for anyone who has dealt with older data, or poorly designed systems when trying to integrate them into larger fabrics.

The ability to both reach deeply to inquire about the most vexing questions and seek to answer the most abstruse problems would be greatly aided by this thinking. Without the care in logically evaluating, and constructing, our future data systems as such, we are destined -- at some relatively far off point -- going to have to come to grips with systematic inaccuracies in our analysis that will not be easily disembroiled.

I suppose that Heidegger and Husserl have shown the vacuity of modern scientists who can destroy the world but have no scientific way of explaining why the world is to be saved. In this I am reminded of Leo Strauss's article on Philosophy as a rigorous science: http://archive.org/stream/LeoStraussphilosophyAsRigorousSciencePoliticalPhilosophy1971/Strauss-PhilosophyAsRigorousScience1971_djvu.txt

The challenge of data or big data is that it pushes us deeper into the cave. We try to make sense of philosophical problems (e.g., "what does it mean?", "Is it good?") with technology. One would be better served by reading Heidegger's Question concerning technology to see that we need to return to the origins of technology so we can understand how to live with it. Yet, even that simple political philosophical question "What is the best way to live" is not a question that is open to a modern scientific answer unless one fundamentally reconsiders science as philosophy rather than science opposed to philosophy. Until we can begin to overcome that false divide there will be no data philosophers properly understood.

Thanks for a stimulating post.

“Hi Jim - going to provide an alternative POV to your assertion. The technologies offer significant new flexibility in handling data processing and availability issues that are recurring / stubborn / hard to address with conventional approaches. The issue isn't if the examples provide have "applicability to your specific business", it is the architecture patterns that matter. Similar to misguided approaches to selecting technologies that only focus on counting the number of TBs, being focused on the application examples rather than what allowed the apps to work is a mistake.”

And I responded:

Thanks for your comment, Tom.

I appreciate, and agree with, your counterpoint to my assertion. The architecture patterns that allow the applications of data science and big data to work is essential to focus on.

However, one example of my concern is the use of big data and data science in the last two U.S. presidential elections. (By the way, I highly recommend to anyone interested in this topic The Victory Lab: The Secret Science of Winning Campaigns an excellent book written by Sasha Issenberg).

A U.S. presidential election is a specialized case with limited applicability to businesses, since it is like selling a product that can only be purchased on one day every four years, and not only that, but only has essentially one competing product to consider (i.e., vote for the Democrat or the Republican).

Furthermore, using data science and big data to target voters is not the same as targeting consumers. Voters are slightly more open to what are essentially aggressive marketing messages because voting is often seen as a civic duty and a way to add your voice to democracy. Consumers are far less open to aggressive marketing messages because no matter how popular a consumer product is (e.g., iPhones and iPads), it is not a civic duty or expression of democracy to purchase one.

Therefore, the use of data science and big data in elections is important to understand as a voter and a citizen (as well as a data privacy advocate), but it doesn't really help the average organization understand how to apply big data to their business activities, so from my perspective it is not a compelling business case.

Best Regards,

Jim

And Tom Deutsch responded:

“Hi Jim - thoughtful discussion, which is great. In the spirit of cooperative disagreement (and it is intended to be cooperative) the election use case above is just a segmentation use case. And that is totally applicable for any business that deals with customers (or suppliers), which covers a lot of ground. The mind set of using all the available data to predict behavior and target those inclined to cooperate (purchase or vote) with you is really what the use case is focused on. It just happens to be expressed in a political use case.”

And I responded:

Hi Tom, thanks for continuing the discussion. I am always open to disagreement, cooperative or otherwise :-)

Yes, the election case is a segmentation use case. But not all segmentation (or other types of) use cases are created equal, nor can they all be extrapolated to other applications.

Predicting behavior is foundational to understanding both voters and consumers, but what works for one doesn't necessarily work for the other. Perhaps I am quibbling over trifle distinctions, but political behavior (at least in the U.S.) is very polarized (as well as often oversimplified to Democrat vs. Republican or Conservative vs. Liberal) whereas consumer behavior is more diversified, which, in a certain sense, makes political behavior more predictable than consumer behavior.

And Tom Deutsch responded:

“Hi Jim - I prefer cooperative, especially give the spirit of our discussion here!

So while understanding your point (and not losing sight of the article being a well placed reminder not to get carried away in the big data hype) the techniques utilized aren't limited to sizing up voters. The data may be specific but the techniques are not, and that makes it applicable to a whole range of organizations.”

And I responded:

Hi Tom, yes cooperative disagreement is always best. In fact, in academic settings, there is a similar concept known as “adversarial collaboration” where two researchers with polarized viewpoints co-write a paper together to provide a more comprehensive coverage of a topic.

I definitely agree with your point about similar techniques, but different data. The techniques — the methodology of true data science — is something that is applicable, as you said, to a whole range of organizations, as well as applications.

My only real fear is that distinction seems too often to get lost in the hype of big data.

It's always the (literal) life-and-death situations where you want 100% quality - data quality, program quality, procedural quality. This is why initiatives such as a national health record - although a worthy ambition - are looked on with suspicion by medicos. Even in hospital, the good staffers will always ask you your name, date of birth, and why you're in there before they do anything. They don't even trust the chart hanging on the end of your bed! - not until they've cross-checked with you, the patient, to ensure the wrong chart hasn't been put there.

“I'm not sure about this. It would be nice to think that we could "spin" things and promote a positive experience ("Hey! Why not come and try our new reports, they taste great!"). But I've found that the question isn't so much about expectations, as about visibility - people don't know what they can trust (because there's no lineage, no reconciliation, no traceability etc etc), so they end up with a default position of not trusting anything.

We had an illustrative case here at the University last week, where one of the Heads of School asked for a report on how many students were due to commence in first year. Two different reports gave two different answers (materially different numbers), but there was no narrative to explain where the numbers had come from, what calculations had been applied, what filters were used to constrain the query, etc.

It turned out when we did the forensic analysis that there were NINE different underlying issues that were contributing to the difference in reported numbers! No visibility = no credibility = no trust = no value.”

And I responded:

Thanks for your comment, Alan.

I agree with you that trying to create an a priori (i.e., not dependent on experience) positive expectation about data is improbable, but I would say that you provided another excellent example of how a negative expectation about data was created.

In your case, I assume that the Head of School who requested data, and then received two conflicting reports, probably now has a negative expectation about student data managed by the University.

So now, you have the even more difficult task of altering an a posteriori (i.e., dependent on experience) negative expectation about data -- in other words, how are you going to convince that Head of School that they should expect a positive experience with data next time?

Best Regards,

Jim

@Dylan — Great point about these types of risks increasing as we move further into a data-centric society.

@Melinda — Excellent point about the quality of machine-generated data raising different kinds of issues. For sensor data, poor quality could be an indication of a poorly calibrated sensor. If that sensor is monitoring oxygen levels on a spacecraft carrying humans, for example, on the journey to Mars, then poor data quality could make the astronauts dead on arrival.

For fraud detection, we had to worry about putting people who really couldn't afford a legal battle through a costly investigation—not to mention wasting precious resource.

For the military and spacecraft applications I'm working on now, data quality is also very important, though because it's machines generating the data, you have to worry about different kinds of issues.

I've experienced this personally when my son's medical data was mixed up with another child in a separate part of the country. We started receiving his medical appointments because the patient ID's had been duplicated. Doesn't take a data quality guru to imagine the implications if physicians are looking at the wrong medical history.

Another example where accurate data is essential is in war-time, I remember the US bombing of the Chinese embassy due to out-of-date information.

Of course there are thousands of other instances where safety can be compromised through bad data, as we move further into a data-centric society the risks increase.

Great post as ever.

Dylan Jones

Community Manager
Data Quality Pro.com