The Once and Future Data Quality Expert
Jim Harris in Associations, Data Quality, Debates tagged Best of 2009, IAIDQ Saturday, November 7, 2009 at 11:11PM Wednesday, November 11 is World Quality Day 2009.
World Quality Day was established by the United Nations in 1990 as a focal point for the quality management profession and as a celebration of the contribution that quality makes to the growth and prosperity of nations and organizations. The goal of World Quality Day is to raise awareness of how quality approaches (including data quality best practices) can have a tangible effect on business success, as well as contribute towards world-wide economic prosperity.
IAIDQ
The International Association for Information and Data Quality (IAIDQ) was chartered in January 2004 and is a not-for-profit, vendor-neutral professional association whose purpose is to create a world-wide community of people who desire to reduce the high costs of low quality information and data by applying sound quality management principles to the processes that create, maintain and deliver data and information.
Since 2007 the IAIDQ has celebrated World Quality Day as a springboard for improvement and a celebration of successes. Please join us to celebrate World Quality Day by participating in our interactive webinar in which the Board of Directors of the IAIDQ will share with you stories and experiences to promote data quality improvements within your organization.
In my recent Data Quality Pro article The Future of Information and Data Quality, I reported on the IAIDQ Ask The Expert Webinar with co-founders Larry English and Tom Redman, two of the industry pioneers for data quality and two of the most well-known data quality experts.
Data Quality Expert
As World Quality Day 2009 approaches, my personal reflections are focused on what the title data quality expert has meant in the past, what it means today, and most important, what it will mean in the future.
With over 15 years of professional services and application development experience, I consider myself to be a data quality expert. However, my experience is paltry by comparison to English, Redman, and other industry luminaries such as David Loshin, to use one additional example from many.
Experience is popularly believed to be the path that separates knowledge from wisdom, which is usually accepted as another way of defining expertise.
Oscar Wilde once wrote that “experience is simply the name we give our mistakes.” I agree. I have found that the sooner I can recognize my mistakes, the sooner I can learn from the lessons they provide, and hopefully prevent myself from making the same mistakes again.
The key is early detection. As I gain experience, I gain an improved ability to more quickly recognize my mistakes and thereby expedite the learning process.
James Joyce wrote that “mistakes are the portals of discovery” and T.S. Eliot wrote that “we must not cease from exploration and the end of all our exploring will be to arrive where we began and to know the place for the first time.”
What I find in the wisdom of these sages is the need to acknowledge the favor our faults do for us. Therefore, although experience is the path that separates knowledge from wisdom, the true wisdom of experience is the wisdom of failure.
As Jonah Lehrer explained: “Becoming an expert just takes time and practice. Once you have developed expertise in a particular area, you have made the requisite mistakes.”
But expertise in any discipline is more than simply an accumulation of mistakes and birthdays. And expertise is not a static state that once achieved, allows you to simply rest on your laurels.
In addition to my real-world experience working on data quality initiatives for my clients, I also read all of the latest books, articles, whitepapers, and blogs, as well as attend as many conferences as possible.
The Times They Are a-Changin'
Much of the discussion that I have heard regarding the future of the data quality profession has been focused on the need for the increased maturity of both practitioners and organizations. Although I do not dispute this need, I am concerned about the apparent lack of attention being paid to how fast the world around us is changing.
Rapid advancements in technology, coupled with the meteoric rise of the Internet and social media (blogs, wikis, Twitter, Facebook, LinkedIn, etc.) has created an amazing medium that is enabling people separated by vast distances and disparate cultures to come together, communicate, and collaborate in ways few would have thought possible just a few decades ago.
I don't believe that it is an exaggeration to state that we are now living in an age where the contrast between the recent past and the near future is greater than perhaps it has ever been in human history. This brave new world has such people and technology in it, that practically every new day brings the possibility of another quantum leap forward.
Although it has been argued by some that the core principles of data quality management are timeless, I must express my doubt. The daunting challenges of dramatically increasing data volumes and the unrelenting progress of cloud computing, software as a service (SaaS), and mobile computing architectures, would appear to be racing toward a high-speed collision with our time-tested (but time-consuming to implement properly) data quality management principles.
The times they are indeed changing and I believe we must stop using terms like Six Sigma and Kaizen as if they were a shibboleth. If these or any other disciplines are to remain relevant, then we must honestly assess them in the harsh and unforgiving light of our brave new world that is seemingly changing faster than the speed of light.
Expertise is not static. Wisdom is not timeless. The only constant is change. For the data quality profession to truly mature, our guiding principles must change with the times, or be relegated to a past that is all too quickly becoming distant.
Share Your Perspectives
In celebration of World Quality Day, please share your perspectives regarding the past, present, and most important, the future of the data quality profession. With apologies to T. H. White, I declare this debate to be about the difference between:
The Once and Future Data Quality Expert
Related Posts
A Portrait of the Data Quality Expert as a Young Idiot
The Nine Circles of Data Quality Hell
Additional IAIDQ Links
IAIDQ Ask The Expert Webinar: World Quality Day 2009
IAIDQ Ask The Expert Webinar with Larry English and Tom Redman
INTERVIEW: Larry English - IAIDQ Co-Founder
INTERVIEW: Tom Redman - IAIDQ Co-Founder
Reader Comments (5)
Jim,
Thanks for a remarkable, brave and well thought post about the future of our profession.
All the best,
Henrik
Thanks for the kind words, Henrik,
As always, your feedback is greatly appreciated and highly valued.
Best Regards,
Jim
Timely post Jim.
I recently attended the DM&IQ conference in London and sat in on a panel that discussed some of the future trends, such as cloud computing. It was a great discussion, highly polarised, and as I came home I thought about how far we've come as a profession but more importantly, how much more there is to do.
The reality is that the world is changing, the volumes of data held by businesses are immense and growing exponentially, our desire for new forms of information delivery insatiable and the opportunities for innovation boundless.
I posted a minor rant/call to arms/checklist about this recently in WANTED: Data Quality Entrepreneurs because I really believe we're not innovating as an industry anything like we should be. The cloud, as an example, offers massive opportunities for a range of data quality services but I've certainly not read anything in the media or press that indicates someone is capitalising on this.
There are a few recent data quality technology innovations which have caught my eye but I also think there is so much more vendors should be doing.
On the personal side of the profession I think online education is where we're headed. The concept of localised training is now being replaced by online learning.
With the Internet you can now train people on every continent so why aren't more people going down this route? Arkady Maydanchik has obviously capitalised on this with his eLearning online curriculum and of course we'll continue to push educational materials on Data Quality Pro so I think if you're a "guru" author, trainer or consultant you need to think of new ways to engage with your clients/trainees using the tools available.
I find it incredibly ironic when I speak to data quality specialists who admit that "they don't have the first clue about all this social media stuff". This is the next generation of information management, it's here right now, they should be embracing it.
What worries me is that the growth of information doesn't match the maturity and growth of our profession. For example, we really need more people who can articulate the value of what we can offer. Ted Friedman made a great point on Twitter recently when he talked about how people should stop moaning about executives that "don't get it" and instead focus on improving ways to demonstrate the value of data quality improvement.
Anyway, rant over, I agree with your points completely, just because we've come a long way doesn't mean we know it all, there is still a hell of a long way to go.
Jim,
Excellent, timely, thought provoking post - well done.
I, like you, have observed the staggering rate of change we are currently experiencing.
Walt Disney said on the Inauguration day of Disney World "If it can be dreamed, it can be done."
These days, when I dream of something, I check Google, and discover it already has been done!
What does the accelerating rate of change mean for the future of Data Quality?
I see a number of factors coming into play that will lead to improved Data Quality, among them are what I call:
1. Ryanair data entry model
2. Plug and Play Data
1. Ryanair data entry model
Ryanair is the largest budget airline in Europe, modeled on Southwest Airlines. They are constantly seeking to improve their processes to take cost out. They were one of the first airlines to move to Internet bookings only, with the passenger performing the data entry process. Giving responsibility for data entry to the person with the greatest interest in the quality of the data, results in very high quality data. I personally remember the first time I booked my own airline ticket. I spent at least an hour checking and rechecking before completing the transaction. On the Ryanair site (as with all such sites today), one cannot enter an 'invalid date', such as 30th February. One must select from a list of valid dates. There is effectively no free format text. One must pay with a credit card, hence one's personal details must match those held by the credit card company.
Conclusion 1: Where feasible - Enterprises should seek to have data entry performed by the person with the greatest interest in the quality of the data.
2. Plug and Play Data
I believe that we in the Data Quality profession need a paradigm shift in the way we think about data. We need to promote a vision of the future in which data is reusable and interchangeable - a world of "Plug and Play Data".
Everybody, including senior management, is familiar with the concepts of "plug and play" and reusable play blocks. The Data Quality Profession needs to educate both Business and IT on the need to create "plug and play data". Senior management need to understand the need for “standard data components”, that can be easily interconnected to satisfy the increasingly dynamic information requirements of the Enterprise. This is particularly important within the Reference Data industry. The reference data industry should lead the way in providing “plug and play data”. This will educate the business world and lead it to strive for data with similar capability.
The incredible advances we are observing on the Internet are facilitated by the global adoption of "plug and play" type standards. We are all familiar with "plug and play devices". We connect our camera to our PC, and the PC immediately recognises the device and starts communicating with it. Global adoption of International standards have facilitated this.
The Semantic Web is putting standards in place that will facilitate the creation and use of Plug and Play Data. We are seeing the start of this with the social media sites you mentioned. Information we post on our blogs effortlessly appears on our LinkedIn profile etc.
However, the social media applications have no understanding of the content of our posts. Understanding the content may not be appropriate for blog posts, which by nature contain free format text, with opinions etc. The Semantic Web will enable applications to understand exactly what is in the content. This will transform data interchange where the content of data fields should conform to an agreed standard.
Conclusion 2: We need to promote a vision of the future in which data is reusable and interchangeable - a world of "Plug and Play Data", facilitated by the Semantic web.
Apologies for the long post...
Thanks again for starting this debate,
Rgds Ken
Great topic, Jim and one to which I'm always interested in contributing my humble piece...
I believe the future of data quality will reflect changing attitudes in the way we treat the environment...instead of largely reactionary, downstream processes we will begin to see data quality pushed further and further upstream. Ken's airline example is a case in point, but there is room to go further. The reason is that, unlike computers, people use data with the need for both precision and ambiguity.
The solution is going to be a hybrid between existing concepts of universal grammar, localized semantics and the symmetry-based geometry which currently accounts for almost every aspect of our physical world.
That last may sound a bit high concept, but the model I'm working on strongly suggests that at the intersection of these three concepts lies a new way of bridging compiler-style precision with fuzzy human connectivity.
John