Jim Harris neatly summarised the complexity of data management problems in his recent post Data Quality and Miracle Exceptions.In the post he makes two key points. Firstly, data quality must be ongoing and sustainable and, secondly, there is no miracle cure – be it a new methodology, a new piece of technology or a new process.
The 80’s Depeche Mode singles hits the nail on the head. In order to successfully deliver data governance (and data quality) you have to get the balance right between methodology, process and technology.
From our experience, business tend to be guided by the vested interests of the particular sales pitch. Consulting driven sales people will punt methodology and hope to sell a lot of bodies. Similarly, the principle that just dropping a tool will resolve the issue is equally misguided.
Any sensible approach must combine methodology and experience with appropriate technology in order to enable a sustainable, ongoing data quality capability within the organisation.
At one client we were pulled into the process right at the end of the evaluation process. They had already budgeted to pull in a large team of SQL programmers to analyse and clean their data. We were able to show that we could achieve the objective with two consultants using a tested methodology and a data quality platform at a fraction of the cost of the alternative.
One other point – the requirement for the project was “to clean our data” – with very little understanding initially as to what this meant. Experience meant that we were able to manage expectations and guide the client through an initial data audit – we were fortunate that they were reasonable and adapted to reality. The biggest challenge that is always faced is that you cannot make firm commitments as to scope and cost before you have profiled the data and linked the results to the actual business need.
The second challenge, as Jim points out, is that data management does not have an end date – as discussed in my post data quality is not a project. It is critical that expectations are managed because any approach that does consider sustainability (through a combination of culture change, process and automation) is ultimately just throwing money away.even tactical projects, such as data migrations provide an excellent opportunity to define the requirement for ongoing data excellence, and the data cleansing process should be reusable in the new system.
this post was first published on the dataqualitymatters blog