The Wisdom of Failure
Jim Harris in Blogs, Books, Data Quality, Interviews tagged Business Intelligence, Philosophy Sunday, July 26, 2009 at 8:13AM Earlier this month, I had the honor of being interviewed by Ajay Ohri on his blog Decision Stats, which is an excellent source of insights on business intelligence and data mining as well as interviews with industry thought leaders and chief evangelists.
One of the questions Ajay asked me during my interview was what methods and habits would I recommend to young analysts just starting in the business intelligence field and part of my response was:
“Don't be afraid to ask questions or admit when you don't know the answers. The only difference between a young analyst just starting out and an expert is that the expert has already made and learned from all the mistakes caused by being afraid to ask questions or admitting when you don't know the answers.”
It is perhaps one of life’s cruelest paradoxes that some lessons simply cannot be taught, but instead have to be learned through the pain of making mistakes. To err is human, but not all humans learn from their errors. In fact, some of us find it extremely difficult to even simply acknowledge when we have made a mistake. This was certainly true for me earlier in my career.
The Wisdom of Crowds
One of my favorite books is The Wisdom of Crowds by James Surowiecki. Before reading it, I admit that I believed crowds were incapable of wisdom and that the best decisions are based on the expert advice of carefully selected individuals. However, Surowiecki wonderfully elucidates the folly of “chasing the expert” and explains the four conditions that characterize wise crowds: diversity of opinion, independent thinking, decentralization and aggregation. The book is also balanced by examining the conditions (e.g. confirmation bias and groupthink) that can commonly undermine the wisdom of crowds. All and all, it is a wonderful discourse on both collective intelligence and collective ignorance with practical advice on how to achieve the former and avoid the latter.
Chasing the Data Quality Expert
Without question, a data quality expert can be an invaluable member of your team. Often an external consultant, a data quality expert can provide extensive experience and best practices from successful implementations. However, regardless of their experience, even with other companies in your industry, every organization and its data is unique. An expert's perspective definitely has merit, but their opinions and advice should not be allowed to dominate the decision making process.
“The more power you give a single individual in the face of complexity,” explains Surowiecki, “the more likely it is that bad decisions will get made.” No one person regardless of their experience and expertise can succeed on their own. According to Surowiecki, the best experts “recognize the limits of their own knowledge and of individual decision making.”
“Success is on the far side of failure”
One of the most common obstacles organizations face with data quality initiatives is that many initial attempts end in failure. Some fail because of lofty expectations, unmanaged scope creep, and the unrealistic perspective that data quality problems can be permanently “fixed” by a one-time project as opposed to needing a sustained program. However, regardless of the reason for the failure, it can negatively affect morale and cause employees to resist participating in the next data quality effort.
Although a common best practice is to perform a post-mortem in order to document the lessons learned, sometimes the stigma of failure persuades an organization to either skip the post-mortem or ignore its findings.
However, in the famous words of IBM founder Thomas J. Watson: “Success is on the far side of failure.”
A failed data quality initiative may have been closer to success than you realize. At the very least, there are important lessons to be learned from the mistakes that were made. The sooner you can recognize your mistakes, the sooner you can mitigate their effects and hopefully prevent them from happening again.
The Wisdom of Failure
In one of my other favorite books, How We Decide, Jonah Lehrer explains:
“The brain always learns the same way, accumulating wisdom through error...there are no shortcuts to this painstaking process...becoming an expert just takes time and practice...once you have developed expertise in a particular area...you have made the requisite mistakes.”
Therefore, although it may be true that experience is the path that separates knowledge from wisdom, I have come to realize that the true wisdom of my experience is the wisdom of failure.
Related Posts
A Portrait of the Data Quality Expert as a Young Idiot
All I Really Need To Know About Data Quality I Learned In Kindergarten
The Nine Circles of Data Quality Hell
Reader Comments (7)
On Twitter, Henrik Liliendahl Sørensen commented:
"To err is human, but to persist is diabolical" - Seneca
If I change the expression data quality to "cooking" or "application development", then the article is still valid. So it contains great general "wisdoms" on how to become an expert from an inexperienced greenhorn.
You have mentioned the necessity of the lessons learned meetings, but in my opinion the progressive learning curve of a team is based on the finding and reusing of the lessons learned. So we should underline once again the using of the learning cycles: kickoff - project execution - lessons learned not to fall in the same trap in two years.
Although a common best practice is to perform a post-mortem in order to document the lessons learned, sometimes the stigma of failure persuades an organization to either skip the post-mortem or ignore its findings.
However, in the famous words of IBM founder Thomas J. Watson: “Success is on the far side of failure.”
So very true. I recommend to organizations that they bite the bullet and just learn from their mistakes. Regardless of the type of project (data-oriented or not), so much can be gleaned from asking what went wrong and why.
Politics, ego, and other less-than-desirable traits often prevent this from happening...
From the LinkedIn Group for The Greater IBM Connection, Vivek Patil commented:
"They say that nothing succeeds like success.
But in reality nothing fails like success as we do not learn from success. It is failures that teach us."
From the LinkedIn Group for TDWI, Wayne Kurtz commented:
"I was about to say 'what a bunch of baloney' then I read your blog post and I can say I've rarely seen a simple set of phrases that makes as much sense to me as a consultant.
So many consultants simply cannot admit they don't know something...
Are they afraid they will loose their gig? What ever happened to 'I don't know but I'll get back to you as soon as I find out?'"
And I responded:
I share your question, especially since I am dubious whenever someone (consultant or otherwise) always has an answer to every question. Sometimes even when they do know an answer, I appreciate it when they check around to see if there are other answers so that we can then discuss our options and make the best decision for the current situation.
From the LinkedIn Group for TDWI, Ajaya Gupta commented:
"True. You only succeed when you reform a failure, otherwise you do not succeed. Wisdom is fool proof."
If you are a DBA, you expect failure. DBAs deal with it by failover, replication, restore, etc. on a daily basis. Dealing with failure and anticipating it is a big part of their job.
As a data modeler/architect, I had to deal with the failure in my earlier designs. I used to to think about unanticipated changes. Now I assume all can change. You eventually learn to make your models as flexible and extensible as possible, without being too abstract. It is easy to explain these kinds of models to developers, etc., because you typically have an answer for most questions. The things you cannot change are suddenly needing to provide history, or past snapshots when none is captured.
Everybody has to be clear on the requirements in the beginning or there is failure at the end.
DBAs and Data modelers are similar in that they both learn from working with experts. I was fortunate enough to learn PL/SQL from experts and then work with some great data modelers.
The biggest thing I have to deal with is the failure before I start. You cannot "fix" a database by reverse engineering a non-modeled database and then tweaking it. You eventually get into design trouble - typically by tacking on columns to tables, using "indicators", or adding "alternate" tables. This to me is a big cost factor in projects. A lot of coding ($) is needed to overcome lack of initial design.
Another failure I learned to overcome was not to cripple your developers. You can make life hard for developers if you model so abstract that developers cannot understand it; do not provide load defaults ("not found") in warehouse reference data; or most importantly, not give developers something to work so they can start writing their code before the model is fully designed. You have to be able to work with them, and they will find design/requirement problems right away if they get a chance to write code early. This goes against the "design first, then code" principle. In reality, people start off running SQL against source data anyway to look for definition problems.