Common Enterprise wide Data Governance Issues #9: Data Migration and ETL projects are Metadata driven

This post is one of a series dealing with common Enterprise Wide Data Governance Issues.  Assess the status of this issue in your Enterprise by clicking here: Data Governance Issue Assessment Process

Too often, Data Migration and ETL projects are built on the basis of Metadata, without measuring what is actually contained in the source data fields.  This happens when the IT function build data ‘pipes’ on the basis of what the metadata says the source fields should contain, and don’t perform data content analysis, or data profiling, to find out what the source fields actually contain.

Impact:
The IT function turn the  ‘tap’ on, the data flows through the ‘pipes’ and the business express surprise, followed by denial, when expectations cannot be met due to data quality issues.  This is known as the ‘Load and Explode’ approach to data.

Solution:
To prevent ‘Load and Explode’ impacting the success of your data dependent projects, agree and apply the following policy:

Before building, or purchasing a system that is dependent on existing data, projects must complete the following process:

  1. Define what data is required.
  2. Define the quality requirements of the required data.
  3. Identify the source of the required data.
  4. Specify the data quality metrics to be captured.
  5. Measure the quality of the available source data.
  6. Understand the implications of the quality of available source data for the proposed system.
  7. If necessary, and if feasible, implement data quality improvement measures to raise the quality to the required level.
  8. Worst case – if the facts tell you data quality is too low and cannot be improved – Cancel the project and save yourself a ton of money!

Your experience:
Have you faced the above issue in your organisation, or while working with clients?  What did you do to resolve it?  Please share your experience by posting a comment – Thank you – Ken.