Know your data

You must know your data.

Do you know what’s in your data box of chocolates?

You must know where it is, what it should contain and what it actually contains.

When your data does not contain what it should, you must have a process for correcting it.

CEOs, CFOs and CROs often take the above as “given”.  They make business critical decisions using information derived from data within their organisation.  After all, its applied common sense.

For the insurance industry, Solvency II requires evidence that you are applying common sense.

If you operate in the EU market or process the personal data of EU data subjects, you must comply with the EU General Data Protection Regulation (GDPR) or face severe fines. To comply, you must “know your (personal) data” and how you manage it.

In my experience, data is like a box of chocolates “You never know what you’re gonna get.”

Do you know your data?

Charter of Data Consumer rights and responsibilities

Time for charter of Data Consumer rights and responsibilities

There are many rights enshrined in law that benefit all of us. One example is the UN Charter of Human Rights.  Another example is the “Consumer Rights” protection most countries enforce to guarantee us, the buying public, the right to expect goods and services that are of good quality and “fit for purpose”.  As buyers of goods and services, we also have responsibilities.  If you or I buy a “Rolex watch” for $10 from a casual street vendor, we cannot claim consumer protection rights if the watch stops working within a week. “Let the buyer beware” or “Caveat Emptor” is the common sense responsibility that we, as consumers must observe.

I have previously written about business users’ right to expect good data plumbing. Business users (of data) have responsibilities also.  I believe its time to agree a charter of rights and responsibilities for them.  Business users of data are “Data Consumers” – people who use data to perform their work, whatever work that may be.  Data Consumers make decisions based on the data or information available to them. Examples can range from a doctor prescribing medication based on the information in a patient’s health records, to a multi-national chief executive deciding to buy a business based on the performance figures available, to an actuary developing an internal model to determine Solvency II Capital Requirements.

What rights and responsibilities should data consumers have?

Here’s my starter set:

  • The right to expect data that is “fit for purpose”, data that is complete, appropriate and accurate.
  • The responsibility to define what “fit for purpose” data means to them.
  • The right to expect guidance and assistance in defining what constitutes complete, appropriate and accurate data for them.
  • The responsibility to explain the impact that “sub-standard” data would have on the work they do.
  • The right to be informed of the actual quality of the data they use.
  • The right to expect controls in place that verify the quality of the data they use meets the standard they require.

What do you think ? Please feedback your suggestions:

How do you collect your data?

Welcome to part 4 of Solvency II Standards for Data Quality – common sense standards for all businesses.

In my last post I highlighted the Solvency II requirement for Data Quality Management processes, which must include:

  • Assessment of the quality of your data
  • Resolution of material problems identified
Have you included plans for data cleansing to resolve material problems identified? Furthermore, have you considered how you plan to prevent the problems recurring? Solvency II requires you to do this, as set out in the following paragraphs of the CEIOPS’ (EIOPA) advice (Consultation Paper 43):

3.36 The assessment of data quality should have due regard to the quality and performance of the channels used to collect, store, process and transmit data…

Your “Data Supply Chain” is the means by which you “Collect, store, process and transmit data…”. You are expected to know your data supply chain, and to manage it effectively.

3.37 If material problems with the verification of the data quality criteria have been identified, the insurer should try to solve them within an appropriate timeframe… and should work towards the improvement of the data collection, storage or other relevant internal processes, so as to ensure the quality of the future data. Those data limitations should be appropriately documented, including a description of how such situations can be remedied and the assignment of responsibilities within the undertaking.

How do you collect your data?

Solvency II mandates Data Governance

Welcome to part 3 of Solvency II Standards for Data Quality – common sense standards for all businesses.

Regardless of the industry you work in, you make critical business decisions based on the information available to you.  You would like to believe the information is accurate.  I suggest the CEIOPS’ standards for “Accuracy”apply to your business, and your industry, just as much as they apply to the insurance industry.  I would welcome your feedback…

The CEIOPS (now renamed EIOPA) advice makes it clear that Solvency II requires you to have Data Governance in place (which CEIOPS / EIOPA refers to as “internal systems and procedures”).   The following sections of the document make this clear:

3.32 In order to ensure on a continuous basis a sufficient quality of the data used in the valuation of technical provisions, the undertaking should have in place internal systems and procedures covering the following areas:

• Data quality management;

• Internal processes on the identification, collection, and processing of data; and

• The role of internal/external auditors and the actuarial function.

3.1.4.1 Data quality management – Internal processes

3.33 Data quality management is a continuous process that should comprise the following steps:

a) Definition of the data;

b) Assessment of the quality of data;

c) Resolution of the material problems identified;

d) Monitoring data quality.

I will explore the above further in my next post.  Meanwhile, what Data Quality Management processes do you have in place?  Do you suffer from common Enterprise-Wide Data Governance Issues?

What does complete appropriate and accurate mean?

Welcome to part 2 of Solvency II Standards for Data Quality – common sense standards for all businesses.

The Solvency II Standards for Data Quality run to 22 pages and provide an excellent substitute to counting sheep if you suffer from insomnia. They are published by The Committee of European Insurance and Occupational Pensions Supervisors (CEIOPS) (now renamed as EIOPA).

Solvency II Data Quality Standards – not as page turning as a Dan Brown novel

I accept that Data Quality Standards cannot aspire to be as page turning as a Dan Brown novel – but plainer English would help.

Anyway – enough  complaining.  As mentioned in part 1, the standards require insurance companies to provide evidence that their Solvency II submissions are based on data that is “as complete, appropriate, and accurate as possible”.  In this post, I will explore what the regulator means by “complete”, “appropriate” and “accurate”.  I will look at the terms in the context of data quality for Solvency II, and will highlight how the same common sense standards apply to all organisations.

APPROPRIATE: “Data is considered appropriate if it is suitable for the intended purpose” (page 19, paragraph 3.62).

Insurance companies must ensure they can provide for insurance claims. Hence, to be “appropriate”, the data must relate to the risks covered, and the value of the capital they have to cover potential claims.  Insurance industry knowledge is required to identify the “appropriate” data, just as Auto Industry knowledge is required to identify data “appropriate” to the Auto industry etc.

COMPLETE: (This one is pretty heavy, but I will include it verbatim, and then seek to simplify – all comments, contributions and dissenting opinions welcome) (page 19, paragraph 3.64)

“Data is considered to be complete if:

  • it allows for the recognition of all the main homogeneous risk groups within the liability portfolio;
  • it has sufficient granularity to allow for the identification of trends and to the full understanding of the behaviour of the underlying risks; and
  • if sufficient historical information is available.”

As I see it, there must be enough data, at a low enough level of detail, to provide a realistic picture of the main types of risks covered. Enough Historical data is also required, since history of past claims provides a basis for estimating the scale of future claims.

As with the term “Appropriate”,  I believe that Insurance industry knowledge is required to identify the data required to ensure that data is “complete”.

ACCURATE: I believe this one is “pure common sense”, and applies to all organisations, across all industries. (page 19, paragraph 3.66)

Data is considered accurate if:

  • it is free from material mistakes, errors and omissions;
  • the recording of information is adequate, performed in a timely manner and is kept consistent across time;
  • a high level of confidence is placed on the data; and
  • the undertaking must be able to demonstrate that it recognises the data set as credible by using it throughout the undertakings operations and decision-making processes.

Update – In October 2013, following an 18 month consultative process, DAMA UK published a white paper explaining 6 primary data quality dimensions.

1. Completeness
2. Uniqueness
3. Timeliness
4. Validity
5. Accuracy
6. Consistency

For more details see my blog post, Major step forward in Data Quality Measurement


Solvency II Standards for Data Quality – common sense standards for all businesses

When you visit your family doctor, you expect him or her to be familiar with your medical history.  You expect the information your doctor keeps about you to be complete, appropriate and accurate. If you’re allergic to penicillin, you expect your family doctor to know about it. Your health and well being depends on it.  Call it applied common sense.

You expect your family doctor to have information about you that is complete appropriate and accurate

In running your business, you make business critical decisions every day.  You base your decisions on the information available to you, information that you would like to be complete, appropriate, and accurate. Call it more applied common sense.

The Solvency II Standards for Data Quality (EIOPA Consultation Paper 43) apply the same common sense.  They require insurance companies to provide evidence that their Solvency II submissions are based on data that is “complete, appropriate, and accurate”.

This is the first of a series of posts in which I plan to explore the “common sense” Solvency II standards for data quality, and I hope you will join in.

What is Solvency II? When you insure your family home, you would like to think your insurance company will still be around, and will have the funds to compensate you in the event of a fire, break-in or other disaster.  Solvency II is seeking to do just that.  Solvency II requires Insurance companies to prove they have enough capital funding to prevent them failing. Given the backdrop of the ongoing world financial crisis, I believe this is a reasonable objective.

As you may have noticed, I am a fan of “common sense”. Think about it… would you knowingly make a business critical decision on the basis of information that you know to be “incomplete, inappropriate or inaccurate”?  I think not.  So, how do you know that your information is “complete, appropriate and accurate”? The standards set out in The Solvency II Standards for Data Quality enable ALL organisations, not just insurance companies, apply the same common sense standards.

As I add posts, I will link to them from here.

In my second post, I explore what exactly the terms “complete”, “appropriate” and “accurate” mean in the context of data quality for Solvency II, and what they mean for all organisations.

My third post explores the need in all organisations for Data Governance, and Data Quality Management – Solvency II actually mandates the need for Data Governance.

In my fourth post I ask “How do you collect your data“.  I ask you this because common sense (and Solvency II) requires that you know how you collect your data, how you process it, and how you assess the quality of the data you collect and process.

My fifth post Data Governance – Did you drop something?  explores the risk associated with data extractions.

Please join in this debate and share your experience…

Data Governance evidence required for Solvency II compliance

Insurance and Re-Insurance companies operating in Europe must comply with Solvency II by 2012.  Highly qualified actuaries and Insurance industry experts will be required to develop models and perform complex calculations to achieve Solvency II compliance.

Solvency II Data Governance

Employing the best actuaries in the world, developing the best algorithms, and building the best models will count for nothing, if the underlying data used in the models is inappropriate, incomplete or inaccurate.  Not only must the underlying data be appropriate, complete and accurate, companies must provide evidence of their Data Governance processes to prove it.

Article 82 of the Solvency II Directive could not be more clear:  “Member States shall ensure that insurance and reinsurance undertakings have internal processes and procedures in place to ensure the appropriateness, completeness and accuracy of the data used in the calculation of their technical provisions.”

Data Governance provides the “processes and procedures to ensure the appropriateness, completeness and accuracy of data”.   For the past 15 years, I have specialised in Data Governance for critical regulatory compliance programmes.

Over a series of posts I explore the common sense data quality standards required for Solvency II compliance.  The same common sense standards that can enable you get the most from your data, regardless of the industry you are in.

Semantic web and data quality

I have been itching to write this post since reading and listening to Phil Simon’s excellent blog post and podcast – Technology Today, #20: David Siegel and The Semantic Web .  If you havn’t listened to Phil’s interview with David Siegel – I recommend you do so now.   I have listened twice so far, and expect to listen many more times.

I have previously discussed Plug and Play Data – The future for Data Quality and I believe that David’s ideas about the “Pull model” and “the power of pull” will help deliver Plug and Play data.

For example, David explains “Semantic” as “Unambiguous”.   As you know, data entry validation against unambiguous business rules can greatly improve data quality.  The semantic web will enable us to embed our clear unambiguous business rules with the data the rules apply to – WOW ! Bring it on !

Show me your Data Quality

Recently Marty Moseley explored how much data governance you should have in a thought provoking post called how taxing is your data governance.

I added the following comment “I agree with you – lightweight governance is the way to go – as you say “just formalized enough” Data Governance  framework, creating “good enough” deliverables to record your decisions, alternatives, precedents, drivers, policies, procedures/processes, business rules, enforcements and metrics – and find them later when you need to invariably make changes – OR WHEN THE REGULATOR asks to see your audit trail.

The fact is that regulators increasingly require evidence of the data quality of the underlying data on which you base your regulatory submissions.  Not just that they require evidence of your data quality management processes, i.e. your Data Governance.

The law now requires evidence of your data quality

For example, in the UK, Deposit Holders are required to build a Single Customer View (SCV) of depositors (UK FSA Single Customer View requirement)  They must not only build an SCV, the regulation states “We… would need to verify that the SCV information being collated and stored by a deposit taker was adequate and fit for purpose”.   Consultation Paper CP 09/3 about the SCV states “As part of their data cleansing, deposit-taking firms would be required to ensure the existence, completeness and accuracy of all data required for each depositor to facilitate fast payout”

A second example affects Insurance companies providing insurance in Europe (this also affects US Insurers operating in Europe).  EU Directive 2009/138/EC (Solvency II) – Article 82 states: “Member States shall ensure that insurance and reinsurance undertakings have internal processes and procedures in place to ensure the appropriateness, completeness and accuracy of the data used in the calculation of their technical provisions.” In other words – the regulator in each member state is required to review and approve the Data Governance processes of insurers.

Failure to comply with Regulatory requirements can prove expensive. Ask Standard Life.  It cost them over £100 million.  I presented details of the Standard Life case at the Information and Data Quality Seminar Series 2010 – Dublin. (For more details, see slides 11 and 12 in my presentation “Achieving Regulatory Compliance – The devil is in the data“.)

Shortly, working together with DataqualityPro, I will publish an E-Book on how to cost-effectively satisfy the (UK FSA Single Customer View requirement).  UK Deposit Holders must submit their SCV implementation plans to the FSA by the end of July 2010, and must complete their SCV by Jan 2011.  Time is running short.

Are you aware of other laws requiring evidence of Data Governance and Data Quality? If so, please share them.

Common Enterprise wide Data Governance Issues – #14. No Enterprise wide Data Model

I was reading David Loshin’s excellent post How Do You Know What Data is Master Data? and I thought “I know – I’ve covered that question in my blog” – but I hadn’t.  So here it is.

Your “Enterprise Wide Data Model” tells you what data is Master Data.

Unfortunately, most organisations lack an Enterprise Wide Data Model. Worse still, there is often little appreciation among senior management of the need for an Enterprise wide Data Model.

Impact:
The absence of a Enterprise wide Data Model makes it difficult for even technical experts to locate data.  The data model would distinguish between Master data and replicas, and would clarify whether the data in the model is currently in place, or planned for.  Without an Enterprise Wide Data Model, data dependent projects (e.g. BASEL II, Anti Money Laundering, Solvency II) must locate data (especially Master Data) from first principles, and face the risk of not finding the data, or identifying inappropriate sources.   New projects dependent on existing data take longer than necessary to complete, and face serious risk of failure.

Solution:
The CIO should define and implement the following Data policy:

An Enterprise wide Data Model will be developed covering critical Enterprise wide data, in accordance with industry best practice.

Time to sing from the same hymn sheet

One notable exception to the norm:
This is not a plug for IBM…. merely an observation based on my experience.

I worked in an IBM development lab in Dublin during the 90’s. At that time IBM developed a “Financial Services Data Model” (FSDM). Dublin was IBM’s “FSDM centre of excellence”. BASEL II turned FSDM into an “Overnight success”- TEN YEARS after it was developed. Organisations that had adopted IBM’s FSDM found it relatively easy to locate the data required by their BASEL II compliance programme.

I forsee a future in which all financial services organisations will use the same data model, including Financial Regulator(s).  “Singing from the same hymn sheet” will make communication far simpler, and less open to misinterpretation.

The lack of an Enterprise Wide Data Model is just one of the many data governance issues that affect organisations today.  Assess the status of this issue in your Enterprise by clicking here:  Data Governance Issue Assessment Process

Does your organisation have an “Enterprise wide Data Model” – if so, how did you achieve it?  Did you build it from scratch, or start with a vendor supplied model? Please share your experience.