How to deliver a Single Customer View

How to deliver a Single Customer View

How to cost effectively deliver a Single Customer View

Many have tried, and many have failed to deliver a “Single Customer View”.  Well now it’s a regulatory requirement – at least for UK Deposit Takers (Banks, Building Societies, etc.).

The requirement to deliver a Single Customer View of eligible deposit holders indirectly affects every man, woman and child in the UK.  Their deposits, large or small, are covered by the UK Deposit Guarantee Scheme.  This scheme played a key role in maintaining confidence in the banking system during the dark days of the world financial crisis.

UK Deposit Takers must not only deliver the required Single Customer View data, they must provide clear evidence of the data quality processes and controls they use to deliver and verify the SCV data.

The deadline for compliance is challenging.  Plans must be submitted to the regulator by July 2010, and the SCV must be built and verified by Jan 2011.

To help UK Deposit Takers, I have written an E-book explaining how to cost effectively deliver a Single Customer View.  You may download this free from the Dataqualitypro website:

While the document specifically addresses the UK Financial Services Requirement for a Single Customer View, the process steps will help anyone planning a major data migration / data population project.

If you are in any doubt about the need for good data quality management processes to deliver any new system (e.g. Single Customer View, Solvency II, etc.), read the excellent Phil Simon interview on Dataqualitypro about why new systems fail.

My interview with Dylan Jones

Dylan Jones of DataQualityPro interviews me about the process I use to assess common Enterprise wide data issues. Use this process to assess the status of data governance within your organisation or that of a client.

Data Quality Pro interview with Ken O'Connor Data Consultant

Russian Gas Pipe and Data Governance

As you know, Russia supplies Gas to many European countries.

What's flowing through your critical data pipelines?

Do you know what’s in your critical data pipelines?

Could you imagine Italy purchasing gas from Russia without checking what exactly was flowing through the pipe?  I’m no expert on gas pipelines, but I know that before completing the agreement to purchase the gas, Italy and Russia would have agreed metrics such as:

  • Volume of Gas
  • Calorific value (Energy content)
  • etc.

So what? What else would one expect?  Applied common sense… yes?

Why is it that such common sense is often lacking in Data Migration and Data Population projects?  Why do some Enterprises continue to perform data population of, and ongoing data entry to, critical data repositories without fully understanding the data they are pumping into the repository?

A simple example involves Date of Birth.  The business ask the IT function to populate Date of Birth in the new AML / BASEL II / CRM / other repository. Some time later, when data population is complete, the business begin to express concerns:

  • “We never realised we had so many customers aged over 100 ???”
  • “I thought we had more Student customers”
  • “How come so many of our customers share the same birthday ?”
  • “These are not the results we expected”
  • etc.

Performing data population on the basis of what the source data “should contain”, without analysing what exactly it does contain is known as ‘Load and Explode’ approach to Data Population.  I cover this Enterprise Wide Data Issue in more detail here.

We in the “Data Governance”, “Data Quality” industry need to educate the business community on the “common sense” parts of data governance, and the need to engage “Data Governance Professionals”  to ensure that “Data Quality Common Sense” is actually applied.

Feedback welcome – Ken

Business Rules Case Study Part I

I would like to start a discussion about Business Rules.  I hope you will join in.  Over a series of posts I plan to explore questions like:

  1. Why are Business Rules necessary?
  2. What exactly is a Business Rule?
  3. What should happen if the data fails a Business Rule?
  4. What controls should you have in place to manage Business Rules?
  5. Where should you look for Business Rules (if your Enterprise has no Master Business Rules Repository)

I will use a case study from an Anti Money Laundering (AML) programme.

In this AML programme, the client selected a “Best in breed AML vendor solution”.   The vendor specified the data required, and the client was responsible for locating the data to populate the new AML repository, and for the quality of the data entered in the repository.

Why are Business Rules necessary?

A standard AML business requirement is the requirement to monitor “Minor Accounts” (accounts held by customers under 18 years of age) for ‘unusual transaction activity’.  This high level requirement would result in a number of more specific business requirements, such as:

“Generate an AML alert when the total value of Cash lodged in a month, to an account held by a minor, exceeds a predefined amount, say EUR5000”

Having  agreed the above business requirement, the vendor asked the client to provide the Business Rule for identifying a ‘Minor Account’.

So:

1. Why are Business Rules necessary?
Business rules are required to distinguish between product types, customer types, car parts etc. etc.  AML systems require business rules in order to apply different alert rules to different account holder types.

AML business staff are AML experts, not business rules experts.  It was unclear who owned the data and it took a long time for the IT department to research the business rule(s) for the vendor.  Q:  How do business users in your enterprise get details of Business Rules?  Do your business users find it difficult to access the data they require?

Let us suppose the Business Rule supplied to the vendor was:
A minor account may be identified as follows:
1. Account Type: Personal
2. Account SubType:  Minor
3. Customer Age:  Less than 18

The age check was required to manage the risk that an account opened when a customer was a Minor was not converted to a Standard Personal account when the customer reached his/her 18th birthday.

So:

2. What exactly is a Business Rule?

A Business rule provides critical details about data, including the ‘business’ name of the field, the business purpose of the field, the values it may hold, the business meaning of each value, and interdependencies with other data.  Let’s explore this a little further:

  1. Business name of the data field(s):
    In the above example, three data fields are used in the Business Rule:
    ‘Account Type’, ‘Account Subtype,’ and ‘Age’ (probably determined from Date of Birth).’
  2. Business purpose of the data field:
    e.g. ‘Account SubType’ is used to identify different account types, such as ‘Minor’, ‘Mature years’ etc.
  3. Permitted values (also known as enumerations):
    e.g. Permitted values for Account Subtype are 101 to 199.
  4. Business meaning of each permitted value:
    e.g. ‘Account SubType’ value 101 means Minor Account
  5. Interdependencies with other data:
    e.g. ‘Account SubType’ depends on ‘Account Type’
    ‘Account SubType’ value 101 means Minor Account, when Account Type is ‘Personal’
  6. Field precedence:
    This defines the order in which the fields should be interrogated
    e.g.  First check Account Type, then Account Sub Type

The AML vendor configured the AML tool to apply the “MINOR” rule when Account Type was personal, Account SubType =101 (Minor), and Customer Age less than 18.

During testing, few alerts were generated on Minor accounts.  From an AML business perspective, the less alerts generated the better, since the workload for the AML staff is dictated by the number of alerts generated.

The AML business area was pleased with the low number of alerts, and the vendor was pleased that the alert worked ‘as specified’.

However, it was common knowledge that Date of Birth was not populated 100% of the time, so what was happening when there was no Date of Birth present?  There was no culture of  data quality measurement in the Enterprise, and no facilities for data profiling. Custom built SQL queries against the new AML repository identified multiple instances in which the actual data failed to conform to the Business Rules.

So:

3. What should happen if the data fails a Business Rule?
In our AML example, what do you think should happen when:
a) Account Subtype is ‘101’ indicating a MINOR account, but the customer is aged over 18?
b) Account Subtype is ‘101’ indicating a MINOR account, but date of birth is not populated for this customer?

Business Rules define what data fields “should” contain.  On this AML programme, as in all real world data programmes, the actual data content did not always match what was expected.

This only became apparent as a result of custom built data profiling.  Based on the actual content of the data, the AML business area had to ask the vendor to implement Exception Rules to handle the non-conforming data.  In an ideal world, the data would have been corrected.  In the real world of “achieve compliance by a given date, or face a regulatory fine”, workarounds are quite normal.

So – what are Exception Rules?
Exception rules define what must happen when an account contains data that fails to comply with a business rule.

This post is already far longer than I had planned – I hope it hasn’t bored you to tears.
In my next post, I will explore:

Please share your experience by posting a comment – Thank you.

Common Enterprise wide Data Governance Issues #9: Data Migration and ETL projects are Metadata driven

This post is one of a series dealing with common Enterprise Wide Data Governance Issues.  Assess the status of this issue in your Enterprise by clicking here: Data Governance Issue Assessment Process

Too often, Data Migration and ETL projects are built on the basis of Metadata, without measuring what is actually contained in the source data fields.  This happens when the IT function build data ‘pipes’ on the basis of what the metadata says the source fields should contain, and don’t perform data content analysis, or data profiling, to find out what the source fields actually contain.

Impact:
The IT function turn the  ‘tap’ on, the data flows through the ‘pipes’ and the business express surprise, followed by denial, when expectations cannot be met due to data quality issues.  This is known as the ‘Load and Explode’ approach to data.

Solution:
To prevent ‘Load and Explode’ impacting the success of your data dependent projects, agree and apply the following policy:

Before building, or purchasing a system that is dependent on existing data, projects must complete the following process:

  1. Define what data is required.
  2. Define the quality requirements of the required data.
  3. Identify the source of the required data.
  4. Specify the data quality metrics to be captured.
  5. Measure the quality of the available source data.
  6. Understand the implications of the quality of available source data for the proposed system.
  7. If necessary, and if feasible, implement data quality improvement measures to raise the quality to the required level.
  8. Worst case – if the facts tell you data quality is too low and cannot be improved – Cancel the project and save yourself a ton of money!

Your experience:
Have you faced the above issue in your organisation, or while working with clients?  What did you do to resolve it?  Please share your experience by posting a comment – Thank you – Ken.

Common Enterprise Wide Data Governance Issues: #7 No SLAs defined for required quality of critical data

This post is one of a series dealing with common Enterprise Wide Data Governance Issues.  Assess the status of this issue in your Enterprise by clicking here:  Data Governance Issue Assessment Process

In some organisations there is a platitute that states: ‘The Business is responsible for the quality of the data’, but…

  • There are no SLAs defined for the required quality of critical data (Master Data)
  • There is no measurement performed of the actual quality of the data
  • The are no processes available to “The Business” to enable them to measure the quality of the data

Impact: Multiple enterprise wide data quality issues.

Solution:
Agree and implement the following policies:

  1. “The business” must be provided with Standard Enterprise wide data quality measurement processes and tools
  2. Business units must regularly measure the quality of critical data, using standard Enterprise wide processes and tools, and must agree SLAs with the users of the data defining the target quality level.
  3. Where necessary, business units must implement data quality improvement measures to meet the quality defined in the agreed SLA.

Your experience:
Have you faced the above issue in your organisation, or while working with clients?  What did you do to resolve it?  Please share your experience by posting a comment – Thank you – Ken.

Process for assessing status of common Enterprise-Wide Data Governance Issues

If you work with data in large enterprises, you will be aware that the data, and the ability of the business to access that data is seldom as “good” as it should be.  But just how “good” or “bad” is it?

This post outlines a process for assessing the status of common Enterprise-Wide Data Governance  issues within your enterprise, or that of a client.  I use it as the basis for my “Data Governance Health Check”.

These issues can impact your ability to deliver the underlying data required for meaningful CRM, Business Intelligence, etc.. More seriously, they can impact your ability to satisfy regulatory compliance demands (e.g. GDPR, BCBS 239, Solvency II, Anti Money Laundering, BASEL II etc.) in a timely cost effective manner.

Do issues like these affect your enterprise?  If not, how have you resolved or prevented them?  Please share your experience by posting a comment.

Common Enterprise-wide data governance issues:

1. Quality of informational data is not as high as desired


2. Quality of data entered by front-end staff is not as high as desired


3. No culture of Data as an ‘asset’ or ‘resource’


4. No clear ownership of data


5. Business Management don’t understand what “Data Quality” means


6. No Enterprise Wide Data Quality Measurement of Data Content


7. No SLAs defined for the required quality level of critical data


8. Accessibility of data is poor


9. Data Migration and ETL projects are Metadata driven


10. No Master repository of Business Rules


11. No ownership of Cross Business Unit Business Rules


12. No Enterprise Wide Data Dictionary


13. Islands of Data

14. No Enterprise Wide Data Model

Explanation of the scale and the process for using it:

There are 6 levels on the scale, starting at level 1, and increasing to level 6.  The higher the score, the better prepared the organisation is to deal with the issue.  The worst case scenario is actually a score of ZERO, which means that management in the enterprise is not even aware that the issue exists.  To assess the actual status of an issue, ask for documentary evidence to illustrate that the Enterprise  has actually reached that level:

Figure 1: Status of a (data governance) issue.

1. Aware Senior Management is aware that the issue exists.e.g. Data Quality is not measured, or measured in ad-hoc manner.#Evidence: Captured in Issues Log or Requirements document.
2. Understands Senior Management fully understands the issue; the impact of not addressing it; options available to address it, complete with the pros and cons of each option.e.g. Issue paper explains the impact of no Data Quality Metrics on downstream data dependent projects etc.Evidence: Issue Paper, Rationale paper or Point of View paper(s).
3. Policy defined Senior Management has a clearly stated policy/strategy identifying the selected option.e.g. Data Quality Measurement must be performed by each Business Unit, using a standard Enterprise Wide Data Quality Measurement process….Evidence: Policy document / Design Principles/ Communications/ education material
4. Process defined The organistaion has a clearly defined process detailing exactly how the policy / strategy will be implemented, which common services / utilities must be used, and exactly how to use them.E.g. The standard Enterprise Wide Data Quality Measurement process will use ‘off the shelf tool X’, to produce a standard set of Data Quality metrics….Each BU must train N staff in the use of the tool.  Training will take place……Evidence: End To End Process documentation / Education and Training material.
5. Infrastructure in place Infrastructure (systems / common services / utilities) needed to implement the process is in place.E.g. ‘off the shelf tool X’ has been licenced and installed Enterprise Wide.  Staff have been trained …Pilots have been run…Evidence: Programme Infrastructure document / Utility user manuals.
6. Governance in place Governance is in place to ensure that the defined policy is implemented in accordance with the defined process.E.g.  The stakeholders are…The Data Steering Enterprise includes the CIO and ….The reporting process is….. The following controls are in place….Evidence: Programme Governance document / Education / completed sign-offs

Your experience:
How do you assess Data Governance within your organisation, or that of a client? Please share your experience by posting a comment – Thank you – Ken.