What is data integrity?

Data integrity means data with maximum accuracy, consistency, and context for confident decision-making.

Of course, the integrity of data is built upon a foundation of data quality – data that is fit for purpose, accurate, and consistent.

But, in today’s interconnected world quality data is not enough. We need to add business context to deliver data that decision-makers can easily understand and trust.

With a foundation of trusted data, businesses can make confident decisions, train machine learning models with integrity, and build new possibilities based on data.

And the benefits are not limited to improved decision-making. IDC Research shows that investing in data integrity helps to drive improvements across a range of business metrics – from increasing customer satisfaction to improving time to market for new products and capabilities, to increased revenue.

Given the benefits, why are many organisations still struggling?

We need to break the habitual behaviour that builds data debt.

Far too often, data management decisions are made based on expedience, rather than on creating value.

Rather than documenting business context, we rely on subject matter experts leaving us at risk of losing our understanding of what data means or what it is used for. At best, uncovering data context becomes a time-consuming and complex process of identifying stakeholders and asking them to share their knowledge – typically again without recording the knowledge.

Technical knowledge is also easily lost. Data engineers and BI developers work at the code level to hack together data pipelines and reports with very little attempt to document the context of what has been developed and why.

 Under time and budget pressures, we sacrifice data integrity for time, pacifying ourselves with the thought that we will document the work once it is complete. Often, this never happens!

These shortcuts add up! Over time, many organisations have lost control of their data landscape, with many systems and processes poorly understood, replicated, and even redundant but still in play. This lack of knowledge breaks trust.

Pay off the data debt

Data integrity means paying off our data debt. Not all at once – this will be overwhelming.

Start by committing to doing better going forward.

Identify critical data elements based on how they are used to support key business processes, reports or regulations, and agree and define minimum data standards for accuracy, completeness, consistency and so on. 

Then, ask yourself, “What is the critical business context that must be recorded to ensure that new data products – such as reports, data sets or machine learning models can be quickly identified and their purpose and scope understood?”

Begin to capture that information centrally – ideally in a business-friendly shared data catalogue that can be easily accessed and searched by other knowledge workers and decision-makers. Make capturing business context part of the job description of data engineers, BI developers and other members of your data team. 

You can then prioritise and begin to work on improvements to existing systems and processes, slowly reducing accumulated debt where it is having the biggest impact.

The journey to data integrity is not linear.

Specific business cases will drive the need for discrete steps and actions – meaning the focus may well shift from one system or data set to another. Again, a centralised data catalog can help to bring context by showing what is being done, why, and where more focus may be required in the future.

Most business use cases require business and IT teams to work closely together to identify the specific data scope that will deliver meaningful business impact. 

When choosing technology it is helpful to identify tools that will accelerate the delivery of solutions for pressing problems, that can plug gaps in the existing data architecture, that can be extended to solve additional problems in the future, and that are easy for business users to adopt in partnership with IT.

Automation is not a silver bullet – without commitment to building better habits it will have limited value. But, assuming that the commitment is there, tools can reduce the workload for each individual, can automatically harvest and bring context to large swaths of data, and can engage the right people, at the right time, reducing the need for meetings and saving time for all.