Why is data quality essential? Reliable data is the foundation of meaningful insights and informed decision-making. Delve into the importance of prioritizing data quality in your organization.

Data quality tools are necessary but not sufficient

In his recent blog post titled “The Role of Technology in Data Quality Management,” Henrik Liliendahl Sørensen challenges the notion that data quality tools are ineffective in solving data quality issues.

Sørensen highlights the tendency of the consulting community to prioritize process implementation while downplaying the importance of tools, whereas tool vendors may overly emphasize the role of tools while neglecting process and people.

The counter-argument

However, John Owens counters this argument by emphasizing that tools alone cannot solve problems if they are not applied appropriately. He suggests that the current shortage of data quality specialists can be attributed to excessive reliance on tools.

The shortage of data quality skills

While Owens’ point regarding the limitations of tools is valid, I disagree with his assertion that the lack of skills is solely due to a dependence on tools. In my view, we have witnessed a decline in data quality skills over the past decade, primarily driven by two key technology trends.

Dilution of the data quality market

Firstly, the data quality market has become diluted as major data quality tool vendors have been acquired by stack vendors that lack a specific focus on data quality. Analysts like Gartner have rightfully observed a shift towards stacks offering basic data quality capabilities alongside other functionalities such as ETL, BI, or MDM. Consequently, passionate data quality professionals have been leaving these organizations, potentially abandoning technology without fully understanding its appropriate usage.

Limited code-based options

Secondly, there is a concerning trend where SQL tools and analytics solutions are being positioned as data quality solutions, leading to confusion between data profiling and data discovery. For instance, the big data community often promotes Python as a viable data quality tool. While code provides some basic capabilities, these limit the potential of data quality and dilute the skills base.

Tools are enablers

As John Owens pointed out, tools are simply enablers and not problem solvers. This is precisely why leading data quality vendors prioritize communication and empowering business and data stewards to participate in addressing data quality issues. Good data quality tools can deliver better and more consistent results in less time compared to technical alternatives or error-prone manual approaches.

Data quality is a specialist skill

Therefore, there is a definite need for effective data quality tools supported by a team dedicated to data quality. Process and approach alone do not suffice in solving the problem. Automation plays a crucial role in enforcing a consistent and relevant application of data quality processes, especially when dealing with large volumes of data. At the same time, tool-independent data quality training provides the foundational skills to understand how to solve data quality problems

Choosing a Data Quality Partner

Companies should seek service providers who possess a combination of data quality knowledge and expertise in implementing appropriate tools. It is important to recognize that not all tools are equal, and the vendor’s proficiency in leveraging the tool is vital in defining the correct process.

What role must business users take for data quality? Learn about the crucial role business users play in ensuring data quality and ownership in this informative article.

How does metadata enhance data quality? Explore the critical success factors for useful metadata in enhancing data quality and relevance in your organization.

In conclusion, the loss of data quality skills can be attributed to various factors, including the dilution of the data quality market and the misalignment of SQL tools and analytics solutions with true data quality needs. However, a well-rounded approach that incorporates both effective tools and skilled professionals is essential for achieving robust data quality.