Data Verification woes

It is ironic that IT efforts to resolve data quality issues can result in worse data quality than before! While the concept of adding validations to fields is not bad, the results can be!

For example, if the payment option selected is credit card, then a database rule may be created that ensures that the Credit Card number field is populated.

Unfortunately, this may simply improve one metric (Completeness) at the expense of another (Accuracy). In our experience this kind of “required” field has very high level levels of duplication (data capturers reuse a known valid value such as 16 zeroes, or an existing credit card number that is known to pass) or junk (values such as N/A).

It is more difficult to identify and remove  these invalid values than it would have been to deal with the empty field. The resulting poor data is used for operational purposes as well as for planning, leading to inaccurate decision making.

I suggest that your data governance team measure compliance to required business standards for specific data attributes (e.g. must have credit card number) and exceptions can be flagged for manual intervention using an automated Data Profiling and Discovery solution. These exceptions should be linked to Performance Contracts meaning that repeat offenders (for data capture) are penalised financially due to poor appraisals.

IT should also assess the options of adding a rules engine, such as a real time data quality solution, to provide more sophisticated checks and balances that can be reused across multiple applications.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.