It is ironic that IT efforts to resolve data quality issues can result in worse data quality than before! While the concept of adding validations to fields is not bad, the results can be!

Poor verification makes data quality worse

For example, if the payment option selected is credit card, then a database rule may be created that ensures that the Credit Card number field is populated.

Unfortunately, this may simply improve one data quality dimension (Completeness) at the expense of another (Accuracy).

In our experience this kind of “required” field has very high level levels of duplication (data capturers reuse a known valid value such as 16 zeroes, or an existing credit card number that is known to pass) or junk (values such as N/A).

It is more difficult to identify and remove  these invalid values than it would have been to deal with the empty field. The resulting poor data is used for operational purposes as well as for planning, leading to inaccurate decision making.

Flag invalid values

I suggest that your data governance team measure compliance to required business standards for specific data attributes (e.g. must have credit card number) and exceptions can be flagged for manual intervention using an automated Data Profiling and Discovery solution. These exceptions should be linked to Performance Contracts meaning that repeat offenders (for data capture) are penalised financially due to poor appraisals.

IT should also assess the options of adding a rules engine, such as a real time data quality management tool, to provide more sophisticated checks and balances that can be reused across multiple applications.

The importance of data standards: Discover seven insights highlighting the significance of data standards in maintaining data integrity and consistency.

How to ensure accurate data entry: Learn techniques for ensuring the accuracy of data entry to maintain data integrity and reliability.