OCDQ Blog

Photo via Flickr (Creative Commons License) by: toettoet

Welcome to the 100th Obsessive-Compulsive Data Quality (OCDQ) blog post!

Absolutely without question, there is no better way to commemorate this milestone other than to also make this the 5th entry in my ongoing series for expressing my gratitude to my readers for their truly commendable comments on my blog posts. 

Commendable Comments

On Will people still read in the future?, Don Frederiksen commented:

“I had an opportunity to study and write about informal learning in the past year and one concept that resonated with me was the notion of Personal Learning Environments (PLE).

In the context of your post, I would regard reading as one element of my PLE, i.e., a method for processing content.  One power of the PLE is that you can control your content, process, objectives, and tools. 

Your PLE will also vary depending on where you are and even with the type of access you have.

For example, I have just spent the last two days without WiFi.  As frustrated as I was, I adapted my PLE based on that scenario.  This morning, I'm sitting in McDonald's drinking coffee but wasn't in a place to watch your video.  (Thank goodness you posted text.)

Even without my current location as a factor, I don't always watch videos or listen to podcasts because I have less control of the content and/or pace.

In regards to your questions, I like books, I read e-books, online content, occasional video, audio books, and Kindle on the iPhone.  Combine these items with TweetDeck, Google Reader, the paper version of the Minneapolis Star Tribune, Amazon, and the Public Library, and you have identified the regular components of my PLE.  To me the tools and process will vary based on my situation.

I also recognize that other people will most likely employ different tools and processes.  The richness of our environment may suggest a decline in reading, but in the end it all comes down to different strokes for different folks.  Everyone motivated to learn can create their own PLE.”

On Shut Your Mouth, Augusto Albeghi (aka Stray__Cat) commented:

“In my opinion, this is a very slippery slope.

This post is true in a world of good-hearted people where everyone wants the best for the team. 

In the real world, the consultant is someone to blame for every problem the project encounters, e.g., they shut their mouth, they'll never be able to stand the critique and will be fired soon.

The better situation is to have expressed a clear recommendation, and if the the customer chooses not to follow it, then the consultant is formally shielded from any form of critique.

The consultant is likely to be caught in no man's land between opposing factions of the project, and must be able to take the right side by a clear statement.  Some customers ask the consultant what's the best thing to do, in order to blame the consultant instead of themselves if something goes wrong.

However, even given all of this, the advice to listen carefully to the customer is still absolutely the #1 lesson that a consultant must learn.”

On OOBE-DQ, Where Are You?, Jill Wanless (aka Sheezaredhead) commented:

“For us, the whole ‘ease of use’ vs. ‘powerful functionality’ debate was included in the business case for the purchase.  We identified the business requirements, included pros and cons of ease of use vs. functionality and made vendor recommendations based on the results of the pros vs. cons vs. requirements.

Also important to note, and included in our business case, was to question if the ease of use requires an intensive effort or costly training program, especially if your goal is to engage business users.

So to answer your questions, I would say if you have your requirements identified, and you do your homework on the benefits/risks/costs of the software, you should have all the information you need to make a logical decision based on the present situation.

Which, of course, will change somewhere down the line as everything does.

And for goodness sake (did I say goodness?), when things do change, always start with identifying the requirements.  Never assume the requirements are the same as they used to be.”

On OOBE-DQ, Where Are You?, Dylan Jones commented:

“I think the most important trend in recent years is where vendors are really starting to understand how data quality workflows should integrate with the knowledge workforce.

I'm seeing several products really get this and create simple user interfaces and functions based on the role of the knowledge worker involved.  These tools have a great balance between usable interface for business specific roles but also a great deal of power features under the bonnet.  That is the software I typically recommend but again it is also about budget, these solutions may be too expensive for some organizations.

There is a danger here though of adding powerful features to knowledge workers who don't fully understand the ramifications of committing those updates to that master customer list.  I still think we'll see IT playing a major role in the data quality process for some time to come, despite the business-focused marketing we're seeing in vendor land.”

On The Dumb and Dumber Guide to Data Quality, Monis Iqbal commented:

“Pretty convincing post for those allergic to long term corrective measures.

And this spawns another question and that is directed towards software developers who come and work on a product/project involving data manipulation and maintaining the quality of the data but aren't that concerned because they did their job of developing the product and then move on to another assignment.

I know I may be repeating the same arguments as presented in your post (Business vs IT) but these developers did care that the project handles data correctly and yet they aren't concerned about quality in the long term, however the person running the business is.

My point is that although data quality can only be achieved when both parties join hands together, I think it is the stakeholder who has to enforce it during all stages of the project lifecycle.”

Thank You

In this brief OCDQ Video, I express my gratitude to all of my readers for helping me reach my 100th blog post.

If you are having trouble viewing this video, then you can watch it on Vimeo by clicking on this link: OCDQ Video

Thanks for your comment

Blogging has made the digital version of my world much smaller and allowed my writing to reach parts of the world it wouldn’t otherwise have been able to reach.  My native language is English, which is also the only language I am fluent in. 

However, with lots of help from both my readers as well as Google Translate, I have been trying to at least learn how to write “Thanks for your comment” in as many languages as possible.

Here is the list (in alphabetical order by language) that I have compiled so far:

• Afrikaans – Dankie vir jou kommentaar
• Croatian – Hvala na komentaru
• Danish – Tak for din kommentar
• Dutch – Bedankt voor je opmerking
• French – Merci pour votre commentaire
• German – Danke für Deine Anmerkung
• Italian – Grazie per il tuo commento
• Norwegian – Takk for din kommentar
• Portuguese – Obrigado pelo seu comentário
• Spanish – Gracias por tu comentario
• Swedish – Tack för din kommentar
• Welsh – Diolch yn fawr am eich sylw chi

Please help continue my education by adding to (or correcting) the above list by posting a comment below.

Related Posts

Commendable Comments (Part 1)

Commendable Comments (Part 2)

Commendable Comments (Part 3)

Commendable Comments (Part 4)

Photo via Flickr (Creative Commons License) by: ella_marie 

Today is Thanksgiving Day, which is a United States holiday with a long and varied history.  The most consistent themes remain family and friends gathering together to share a large meal and express their gratitude.

This is the fourth entry in my ongoing series for expressing my gratitude to my readers for their truly commendable comments on my blog posts.  Receiving comments is the most rewarding aspect of my blogging experience.  Although I am truly grateful to all of my readers, I am most grateful to my commenting readers. 

Commendable Comments

On Days Without A Data Quality Issue, Steve Sarsfield commented:

“Data quality issues probably occur on some scale in most companies every day.  As long as you qualify what is and isn't a data quality issue, this gets back to what the company thinks is an acceptable level of data quality.

I've always advocated aggregating data quality scores to form business metrics.  For example, what data quality metrics would you combine to ensure that customers can always be contacted in case of an upgrade, recall or new product offering?  If you track the aggregation, it gives you more of a business feel.”

On Customer Incognita, Daragh O Brien commented:

“Back when I was with the phone company I was (by default) the guardian of the definition of a 'Customer'.  Basically I think they asked for volunteers to step forward and I was busy tying my shoelace when the other 11,000 people in the company as one entity took a large step backwards.

I found that the best way to get a definition of a customer was to lock the relevant stakeholders in a room and keep asking 'What' and 'Why'. 

My 'data modeling' methodology was simple.  Find out what the things were that were important to the business operation, define each thing in English without a reference to itself, and then we played the 'Yes/No Game Show' to figure out how that entity linked to other things and what the attributes of that thing were.

Much to IT's confusion, I insisted that the definition needed to be a living thing, not carved in two stone tablets we'd lug down from on top of the mountain. 

However, because of the approach that had been taken we found that when new requirements were raised (27 from one stakeholder), the model accommodated all of them either through an expansion of a description or the addition of a piece of reference data to part of the model.

Fast-forward a few months from the modeling exercise.  I was asked by IT to demo the model to a newly acquired subsidiary.  It was a significantly different business.  I played the 'Yes/No Game Show' with them for a day.  The model fitted their needs with just a minor tweak. 

The IT team from the subsidiary wanted to know how had I gone about normalizing the data to come up with the model, which is kind of like cutting up a perfectly good apple pie to find out how what an apple is and how to make pastry.

What I found about the 'Yes/No Game Show' approach was that it made people open up their thinking a bit, but it took some discipline and perseverance on my part to keep asking what and why.  Luckily, having spent most of the previous few years trying to get these people to think seriously about data quality they already thought I was a moron so they were accommodating to me.

A key learning for me out of the whole thing is that, even if you are doing a data management exercise for a part of a larger business, you need to approach it in a way that can be evolved and continuously improved to ensure quality across the entire organization. 

Also, it highlighted the fallacy of assuming that a company can only have one kind of customer.”

On The Once and Future Data Quality Expert, Dylan Jones commented:

“I recently attended a conference and sat in on a panel that discussed some of the future trends, such as cloud computing.  It was a great discussion, highly polarized, and as I came home I thought about how far we've come as a profession but more importantly, how much more there is to do.

The reality is that the world is changing, the volumes of data held by businesses are immense and growing exponentially, our desire for new forms of information delivery insatiable, and the opportunities for innovation boundless.

I really believe we're not innovating as an industry anything like we should be.  The cloud, as an example, offers massive opportunities for a range of data quality services but I've certainly not read anything in the media or press that indicates someone is capitalizing on this.

There are a few recent data quality technology innovations which have caught my eye, but I also think there is so much more vendors should be doing.

On the personal side of the profession, I think online education is where we're headed.  The concept of localized training is now being replaced by online learning.  With the Internet you can now train people on every continent, so why aren't more people going down this route?

I find it incredibly ironic when I speak to data quality specialists who admit that 'they don't have the first clue about all this social media stuff.'  This is the next generation of information management, it's here right now, they should be embracing it.  I think if you're a 'guru' author, trainer or consultant you need to think of new ways to engage with your clients/trainees using the tools available.

What worries me is that the growth of information doesn't match the maturity and growth of our profession.  For example, we really need more people who can articulate the value of what we can offer. 

Ted Friedman made a great point on Twitter recently when he talked about how people should stop moaning about executives that 'don't get it' and instead focus on improving ways to demonstrate the value of data quality improvement.

Just because we've come a long way doesn't mean we know it all, there is still a hell of a long way to go.”

Thanks for giving your comments

Thank you very much for giving your comments and sharing your perspectives with our collablogaunity.  Since there have been so many commendable comments, please don't be offended if your commendable comment hasn't been featured yet. 

Please keep on commenting and stay tuned for future entries in the series. 

Related Posts

Commendable Comments (Part 1)

Commendable Comments (Part 2)

Commendable Comments (Part 3)

In a July 2008 blog post on Men with Pens (one of the Top 10 Blogs for Writers 2009), James Chartrand explained:

“Comment sections are communities strengthened by people.”

“Building a blog community creates a festival of people” where everyone can, as Chartrand explained, “speak up with great care and attention, sharing thoughts and views while openly accepting differing opinions.”

I agree with James (and not just because of his cool first name) – my goal for this blog is to foster an environment in which a diversity of viewpoints is freely shared without bias.  Everyone is invited to get involved in the discussion and have an opportunity to hear what others have to offer.  This blog's comment section has become a community strengthened by your contributions.

This is the third entry in my ongoing series celebrating my heroes – my readers.

Commendable Comments

On The Fragility of Knowledge, Andy Lunn commented:

“In my field of Software Development, you simply cannot rest and rely on what you know.  The technology you master today will almost certainly evolve over time and this can catch you out.  There's no point being an expert in something no one wants any more!  This is not always the case, but don't forget to come up for air and look around for what's changing.

I've lost count of the number of organizations I've seen who have stuck with a technology that was fresh 15 years ago and a huge stagnant pot of data, who are now scrambling to come up to speed with what their customers expect.  Throwing endless piles of cash at the problem, hoping to catch up.

What am I getting at?  The secret I've learned is to adapt.  This doesn't mean jump on every new fad immediately, but be aware of it.  Follow what's trending, where the collective thinking is heading and most importantly, what do your customers want?

I just wish more organizations would think like this and realize that the systems they create, the data they hold, and the customers they have are in a constant state of flux.  They are all projects that need care and attention.  All subject to change, there's no getting away from it, but small, well planned changes are a lot less painful, trust me.”

On DQ-Tip: “Data quality is primarily about context not accuracy...”, Stephen Simmonds commented:

“I have to agree with Rick about data quality being in the eye of the beholder – and with Henrik on the several dimensions of quality.

A theme I often return to is 'what does the business want/expect from data?' – and when you hear them talk about quality, it's not just an issue of accuracy.  The business stakeholder cares – more than many seem to notice – about a number of other issues that are squarely BI concerns:

– Timeliness ('WHEN I want it')
– Format ('how I want to SEE it') – visualization, delivery channels
– Usability ('how I want to then make USE of it') – being able to extract information from a report (say) for other purposes
– Relevance ('I want HIGHLIGHTED the information that is meaningful to me')

And so on.  Yes, accuracy is important, and it messes up your effectiveness when delivering inaccurate information.  But that's not the only thing a business stakeholder can raise when discussing issues of quality.  A report can be rejected as poor quality if it doesn't adequately meet business needs in a far more general sense.  That is the constant challenge for a BI professional.”

On Mistake Driven Learning, Ken O'Connor commented:

“There is a Chinese proverb that says:

'Tell me and I'll forget; Show me and I may remember; Involve me and I'll understand.'

I have found the above to be very true, especially when seeking to brief a large team on a new policy or process.  Interaction with the audience generates involvement and a better understanding.

The challenge facing books, whitepapers, blog posts etc. is that they usually 'Tell us,' they often 'Show us,' but they seldom 'Involve us.'

Hence, we struggle to remember, and struggle even more to understand.  We learn best by 'doing' and by making mistakes.”

You Are Awesome

Thank you very much for your comments.  For me, the best part of blogging is the dialogue and discussion provided by interactions with my readers.  Since there have been so many commendable comments, please don't be offended if your commendable comment hasn't been featured yet.  Please keep on commenting and stay tuned for future entries in the series.

By the way, even if you have never posted a comment on my blog, you are still awesome — feel free to tell everyone I said so.

Related Posts

Commendable Comments (Part 1)

Commendable Comments (Part 2)

In a recent guest post on ProBlogger, Josh Hanagarne “quoted” Jane Austen:

“It is a truth universally acknowledged, that a blogger in possession of a good domain must be in want of some worthwhile comments.”

“The most rewarding thing has been that comments,” explained Hanagarne, “led to me meeting some great people I possibly never would have known otherwise.”  I wholeheartedly echo that sentiment. 

This is the second entry in my ongoing series celebrating my heroes – my readers.

Commendable Comments

Proving that comments are the best part of blogging, on The Data-Information Continuum, Diane Neville commented:

“This article is intriguing. I would add more still.

A most significant quote:  'Data could be considered a constant while Information is a variable that redefines data for each specific use.'

This tells us that Information draws from a snapshot of a Data store.  I would state further that the very Information [specification] is – in itself – a snapshot.

The earlier quote continues:  'Data is not truly a constant since it is constantly changing.'

Similarly, it is a business reality that 'Information is not truly a constant since it is constantly changing.'

The article points out that 'The Data-Information Continuum' implies a many-to-many relationship between the two.  This is a sensible CONCEPTUAL model.

Enterprise Architecture is concerned as well with its responsibility for application quality in service to each Business Unit/Initiative.

For example, in the interest of quality design in Application Architecture, an additional LOGICAL model must be maintained between a then-current Information requirement and the particular Data (snapshots) from which it draws.  [Snapshot: generally understood as captured and frozen – and uneditable – at a particular point in time.]  Simply put, Information Snapshots have a PARENT RELATIONSHIP to the Data Snapshots from which they draw.

Analyzing this further, refer to this further piece of quoted wisdom (from section 'Subjective Information Quality'):  '...business units and initiatives must begin defining their Information...by using...Data...as a foundation...necessary for the day-to-day operation of each business unit and initiative.'

From logically-related snapshots of Information to the Data from which it draws, we can see from this quote that yet another PARENT/CHILD relationship exists...that from Business Unit/Initiative Snapshots to the Information Snapshots that implement whatever goals are the order of the day.  But days change.

If it is true that 'Data is not truly a constant since it is constantly changing,' and if we can agree that Information is not truly a constant either, then we can agree to take a rational and profitable leap to the truth that neither is a Business Unit/Initiative...since these undergo change as well, though they represent more slowly-changing dimensions.

Enterprises have an increasing responsibility for regulatory/compliance/archival systems that will qualitatively reproduce the ENTIRE snapshot of a particular operational transaction at any given point in time.

Thus, the Enterprise Architecture function has before it a daunting task:  to devise a holistic process that can SEAMLESSLY model the correct relationship of snapshots between Data (grandchild), Information (parent) and Business Unit/Initiative (grandparent).

There need be no conversion programs or redundant, throw-away data structures contrived to bridge the present gap.  The ability to capture the activities resulting from the undeniable point-in-time hierarchy among these entities is where tremendous opportunities lie.”

On Missed It By That Much, Vish Agashe commented:

“My favorite quote is 'Instead of focusing on the exceptions – focus on the improvements.'

I think that it is really important to define incremental goals for data quality projects and track the progress through percentage improvement over a period of time.

I think it is also important to manage the expectations that the goal is not necessarily to reach 100% (which will be extremely difficult if not impossible) clean data but the goal is to make progress to a point where the purpose for cleaning the data can be achieved in much better way than had the original data been used.

For example, if marketing wanted to use the contact data to create a campaign for those contacts which have a certain ERP system installed on-site.  But if the ERP information on the contact database is not clean (it is free text, in some cases it is absent etc...) then any campaign run on this data will reach only X% contacts at best (assuming only X% of contacts have ERP which is clean)...if the data quality project is undertaken to clean this data, one needs to look at progress in terms of % improvement.  How many contacts now have their ERP field cleaned and legible compared to when we started etc...and a reasonable goal needs to be set based on how much marketing and IT is willing to invest in these issues (which in turn could be based on ROI of the campaign based on increased outreach).”

Proving that my readers are way smarter than I am, on The General Theory of Data Quality, John O'Gorman commented:

“My theory of the data, information, knowledge continuum is more closely related to the element, compound, protein, structure arc.

In my world, there is no such thing as 'bad' data, just as there is no 'bad' elements.  Data is either useful or not: the larger the audience that agrees that a string is representative of something they can use, the more that string will be of value to me.

By dint of its existence in the world of human communication and in keeping with my theory, I can assign every piece of data to one of a fixed number of classes, each with characteristics of their own, just like elements in the periodic table.  And, just like the periodic table, those characteristics do not change.  The same 109 usable elements in the periodic table are found and are consistent throughout the universe, and our ability to understand that universe is based on that stability.

Information is simply data in a given context, like a molecule of carbon in flour.  The carbon retains all of its characteristics but the combination with other elements allows it to partake in a whole class of organic behavior. This is similar to the word 'practical' occurring in a sentence: Jim is a practical person or the letter 'p' in the last two words.

Where the analogue bends a bit is a cause of a lot of information management pain, but can be rectified with a slight change in perspective.  Computers (and almost all indexes) have a hard time with homographs: strings that are identical but that mean different things.  By creating fixed and persistent categories of data, my model suffers no such pain.

Take the word 'flies' in the following: 'Time flies like an arrow.' and 'Fruit flies like a pear.'  The data 'flies' can be permanently assigned to two different places, and their use determines which instance is relevant in the context of the sentence.  One instance is a verb, the other a plural noun.

Knowledge, in my opinion, is the ability to recognize, predict and synthesize patterns of information for past, present and future use, and more importantly to effectively communicate those patterns in one or more contexts to one or more audiences.

On one level, the model for information management that I use makes no apparent distinction between the data: we all use nouns, adjectives, verbs and sometimes scalar objects to communicate.  We may compress those into extremely compact concepts but they can all be unraveled to get at elemental components. At another level every distinction is made to insure precision.

The difference between information and knowledge is experiential and since experience is an accumulative construct, knowledge can be layered to appeal to common knowledge, special knowledge and unique knowledge.

Common being the most easily taught and widely applied; Special being related to one or more disciplines and/or special functions; and, Unique to individuals who have their own elevated understanding of the world and so have a need for compact and purpose-built semantic structures.

Going back to the analogue, knowledge is equivalent to the creation by certain proteins of cartilage, the use to which that cartilage is put throughout a body, and the specific shape of the cartilage that forms my nose as unique from the one on my wife's face.

To me, the most important part of the model is at the element level.  If I can convince a group of people to use a fixed set of elemental categories and to reference those categories when they create information, it's amazing how much tension disappears in the design, creation and deployment of knowledge.”

Tá mé buíoch díot

Daragh O Brien recently taught me the Irish Gaelic phrase Tá mé buíoch díot, which translates as I am grateful to you.

I am very grateful to all of my readers.  Since there have been so many commendable comments, please don't be offended if your commendable comment hasn't been featured yet.  Please keep on commenting and stay tuned for future entries in the series.

Related Posts

Commendable Comments (Part 1)

Commendable Comments (Part 3)