Charter of Data Consumer rights and responsibilities

Time for charter of Data Consumer rights and responsibilities

There are many rights enshrined in law that benefit all of us. One example is the UN Charter of Human Rights.  Another example is the “Consumer Rights” protection most countries enforce to guarantee us, the buying public, the right to expect goods and services that are of good quality and “fit for purpose”.  As buyers of goods and services, we also have responsibilities.  If you or I buy a “Rolex watch” for $10 from a casual street vendor, we cannot claim consumer protection rights if the watch stops working within a week. “Let the buyer beware” or “Caveat Emptor” is the common sense responsibility that we, as consumers must observe.

I have previously written about business users’ right to expect good data plumbing. Business users (of data) have responsibilities also.  I believe its time to agree a charter of rights and responsibilities for them.  Business users of data are “Data Consumers” – people who use data to perform their work, whatever work that may be.  Data Consumers make decisions based on the data or information available to them. Examples can range from a doctor prescribing medication based on the information in a patient’s health records, to a multi-national chief executive deciding to buy a business based on the performance figures available, to an actuary developing an internal model to determine Solvency II Capital Requirements.

What rights and responsibilities should data consumers have?

Here’s my starter set:

  • The right to expect data that is “fit for purpose”, data that is complete, appropriate and accurate.
  • The responsibility to define what “fit for purpose” data means to them.
  • The right to expect guidance and assistance in defining what constitutes complete, appropriate and accurate data for them.
  • The responsibility to explain the impact that “sub-standard” data would have on the work they do.
  • The right to be informed of the actual quality of the data they use.
  • The right to expect controls in place that verify the quality of the data they use meets the standard they require.

What do you think ? Please feedback your suggestions:

How to deal with Gobbledygook requirements

In my last post I had a bit of a rant about the Gobbledygook “eligibility requirements” provided by the UK Financial Services Compensation Scheme.

The reality is that business requirements can come from many places, they are often vague, and often overly complex.  They are often imposed on you from outside, as in the case of regulatory requirements, like the UK regulatory requirement to deliver a Single Customer View.

So… life can be tough – you have to get on with it, and deal with “less than perfect” requirements.

Well defined requirements are clear, measurable and testable.  You cannot expect business experts to be expert in defining requirements. As a Data Quality professional, one of your roles is to work with business experts to clarify and simplify their requirements.

Let us look at the “eligibility requirements” provided by the UK Financial Services Compensation Scheme

In essence, some customer types are eligible for compensation, while others are not.  You must use your “parsing skills” to parse the overly complex rules – to “sort the apples from the oranges” so to speak.   Start by listing unique customer types, which include:

  • Sole Trader
  • Credit Union
  • Collective Investment Scheme
  • Trustee of a Collective Investment Scheme
  • Operator of a Collective Investment Scheme

Having done that, you can begin the task of finding out whether you can currently identify these customer types within your data.

The above is just a starting point, I hope it helps.

Feedback welcome, as always.

How do you collect your data?

Welcome to part 4 of Solvency II Standards for Data Quality – common sense standards for all businesses.

In my last post I highlighted the Solvency II requirement for Data Quality Management processes, which must include:

  • Assessment of the quality of your data
  • Resolution of material problems identified
Have you included plans for data cleansing to resolve material problems identified? Furthermore, have you considered how you plan to prevent the problems recurring? Solvency II requires you to do this, as set out in the following paragraphs of the CEIOPS’ (EIOPA) advice (Consultation Paper 43):

3.36 The assessment of data quality should have due regard to the quality and performance of the channels used to collect, store, process and transmit data…

Your “Data Supply Chain” is the means by which you “Collect, store, process and transmit data…”. You are expected to know your data supply chain, and to manage it effectively.

3.37 If material problems with the verification of the data quality criteria have been identified, the insurer should try to solve them within an appropriate timeframe… and should work towards the improvement of the data collection, storage or other relevant internal processes, so as to ensure the quality of the future data. Those data limitations should be appropriately documented, including a description of how such situations can be remedied and the assignment of responsibilities within the undertaking.

How do you collect your data?

Russian Gas Pipe and Data Governance

As you know, Russia supplies Gas to many European countries.

What's flowing through your critical data pipelines?

Do you know what’s in your critical data pipelines?

Could you imagine Italy purchasing gas from Russia without checking what exactly was flowing through the pipe?  I’m no expert on gas pipelines, but I know that before completing the agreement to purchase the gas, Italy and Russia would have agreed metrics such as:

  • Volume of Gas
  • Calorific value (Energy content)
  • etc.

So what? What else would one expect?  Applied common sense… yes?

Why is it that such common sense is often lacking in Data Migration and Data Population projects?  Why do some Enterprises continue to perform data population of, and ongoing data entry to, critical data repositories without fully understanding the data they are pumping into the repository?

A simple example involves Date of Birth.  The business ask the IT function to populate Date of Birth in the new AML / BASEL II / CRM / other repository. Some time later, when data population is complete, the business begin to express concerns:

  • “We never realised we had so many customers aged over 100 ???”
  • “I thought we had more Student customers”
  • “How come so many of our customers share the same birthday ?”
  • “These are not the results we expected”
  • etc.

Performing data population on the basis of what the source data “should contain”, without analysing what exactly it does contain is known as ‘Load and Explode’ approach to Data Population.  I cover this Enterprise Wide Data Issue in more detail here.

We in the “Data Governance”, “Data Quality” industry need to educate the business community on the “common sense” parts of data governance, and the need to engage “Data Governance Professionals”  to ensure that “Data Quality Common Sense” is actually applied.

Feedback welcome – Ken

Business Rules Case Study Part I

I would like to start a discussion about Business Rules.  I hope you will join in.  Over a series of posts I plan to explore questions like:

  1. Why are Business Rules necessary?
  2. What exactly is a Business Rule?
  3. What should happen if the data fails a Business Rule?
  4. What controls should you have in place to manage Business Rules?
  5. Where should you look for Business Rules (if your Enterprise has no Master Business Rules Repository)

I will use a case study from an Anti Money Laundering (AML) programme.

In this AML programme, the client selected a “Best in breed AML vendor solution”.   The vendor specified the data required, and the client was responsible for locating the data to populate the new AML repository, and for the quality of the data entered in the repository.

Why are Business Rules necessary?

A standard AML business requirement is the requirement to monitor “Minor Accounts” (accounts held by customers under 18 years of age) for ‘unusual transaction activity’.  This high level requirement would result in a number of more specific business requirements, such as:

“Generate an AML alert when the total value of Cash lodged in a month, to an account held by a minor, exceeds a predefined amount, say EUR5000”

Having  agreed the above business requirement, the vendor asked the client to provide the Business Rule for identifying a ‘Minor Account’.

So:

1. Why are Business Rules necessary?
Business rules are required to distinguish between product types, customer types, car parts etc. etc.  AML systems require business rules in order to apply different alert rules to different account holder types.

AML business staff are AML experts, not business rules experts.  It was unclear who owned the data and it took a long time for the IT department to research the business rule(s) for the vendor.  Q:  How do business users in your enterprise get details of Business Rules?  Do your business users find it difficult to access the data they require?

Let us suppose the Business Rule supplied to the vendor was:
A minor account may be identified as follows:
1. Account Type: Personal
2. Account SubType:  Minor
3. Customer Age:  Less than 18

The age check was required to manage the risk that an account opened when a customer was a Minor was not converted to a Standard Personal account when the customer reached his/her 18th birthday.

So:

2. What exactly is a Business Rule?

A Business rule provides critical details about data, including the ‘business’ name of the field, the business purpose of the field, the values it may hold, the business meaning of each value, and interdependencies with other data.  Let’s explore this a little further:

  1. Business name of the data field(s):
    In the above example, three data fields are used in the Business Rule:
    ‘Account Type’, ‘Account Subtype,’ and ‘Age’ (probably determined from Date of Birth).’
  2. Business purpose of the data field:
    e.g. ‘Account SubType’ is used to identify different account types, such as ‘Minor’, ‘Mature years’ etc.
  3. Permitted values:
    e.g. Permitted values for Account Subtype are 101 to 199.
  4. Business meaning of each permitted value:
    e.g. ‘Account SubType’ value 101 means Minor Account
  5. Interdependencies with other data:
    e.g. ‘Account SubType’ depends on ‘Account Type’
    ‘Account SubType’ value 101 means Minor Account, when Account Type is ‘Personal’
  6. Field precedence:
    This defines the order in which the fields should be interrogated
    e.g.  First check Account Type, then Account Sub Type

The AML vendor configured the AML tool to apply the “MINOR” rule when Account Type was personal, Account SubType =101 (Minor), and Customer Age less than 18.

During testing, few alerts were generated on Minor accounts.  From an AML business perspective, the less alerts generated the better, since the workload for the AML staff is dictated by the number of alerts generated.

The AML business area was pleased with the low number of alerts, and the vendor was pleased that the alert worked ‘as specified’.

However, it was common knowledge that Date of Birth was not populated 100% of the time, so what was happening when there was no Date of Birth present?  There was no culture of  data quality measurement in the Enterprise, and no facilities for data profiling. Custom built SQL queries against the new AML repository identified multiple instances in which the actual data failed to conform to the Business Rules.

So:

3. What should happen if the data fails a Business Rule?
In our AML example, what do you think should happen when:
a) Account Subtype is ‘101’ indicating a MINOR account, but the customer is aged over 18?
b) Account Subtype is ‘101’ indicating a MINOR account, but date of birth is not populated for this customer?

Business Rules define what data fields “should” contain.  On this AML programme, as in all real world data programmes, the actual data content did not always match what was expected.

This only became apparent as a result of custom built data profiling.  Based on the actual content of the data, the AML business area had to ask the vendor to implement Exception Rules to handle the non-conforming data.  In an ideal world, the data would have been corrected.  In the real world of “achieve compliance by a given date, or face a regulatory fine”, workarounds are quite normal. 

So – what are Exception Rules?
Exception rules define what must happen when an account contains data that fails to comply with a business rule.

This post is already far longer than I had planned – I hope it hasn’t bored you to tears.
In my next post, I will explore:

Please share your experience by posting a comment – Thank you.

Common Enterprise wide Data Governance Issues #9: Data Migration and ETL projects are Metadata driven

This post is one of a series dealing with common Enterprise Wide Data Governance Issues.  Assess the status of this issue in your Enterprise by clicking here: Data Governance Issue Assessment Process

Too often, Data Migration and ETL projects are built on the basis of Metadata, without measuring what is actually contained in the source data fields.  This happens when the IT function build data ‘pipes’ on the basis of what the metadata says the source fields should contain, and don’t perform data content analysis, or data profiling, to find out what the source fields actually contain.

Impact:
The IT function turn the  ‘tap’ on, the data flows through the ‘pipes’ and the business express surprise, followed by denial, when expectations cannot be met due to data quality issues.  This is known as the ‘Load and Explode’ approach to data.

Solution:
To prevent ‘Load and Explode’ impacting the success of your data dependent projects, agree and apply the following policy:

Before building, or purchasing a system that is dependent on existing data, projects must complete the following process:

  1. Define what data is required.
  2. Define the quality requirements of the required data.
  3. Identify the source of the required data.
  4. Specify the data quality metrics to be captured.
  5. Measure the quality of the available source data.
  6. Understand the implications of the quality of available source data for the proposed system.
  7. If necessary, and if feasible, implement data quality improvement measures to raise the quality to the required level.
  8. Worst case – if the facts tell you data quality is too low and cannot be improved – Cancel the project and save yourself a ton of money!

Your experience:
Have you faced the above issue in your organisation, or while working with clients?  What did you do to resolve it?  Please share your experience by posting a comment – Thank you – Ken.

Common Enterprise Wide Data Governance Issues: #7 No SLAs defined for required quality of critical data

This post is one of a series dealing with common Enterprise Wide Data Governance Issues.  Assess the status of this issue in your Enterprise by clicking here:  Data Governance Issue Assessment Process

In some organisations there is a platitute that states: ‘The Business is responsible for the quality of the data’, but…

  • There are no SLAs defined for the required quality of critical data (Master Data)
  • There is no measurement performed of the actual quality of the data
  • The are no processes available to “The Business” to enable them to measure the quality of the data

Impact: Multiple enterprise wide data quality issues.

Solution:
Agree and implement the following policies:

  1. “The business” must be provided with Standard Enterprise wide data quality measurement processes and tools
  2. Business units must regularly measure the quality of critical data, using standard Enterprise wide processes and tools, and must agree SLAs with the users of the data defining the target quality level.
  3. Where necessary, business units must implement data quality improvement measures to meet the quality defined in the agreed SLA.

Your experience:
Have you faced the above issue in your organisation, or while working with clients?  What did you do to resolve it?  Please share your experience by posting a comment – Thank you – Ken.