Couch too big

Is data migration like moving house?

Would you move into a new house without checking that it would fit your furniture? Many of us had the experience of trying to fit a comfy couch, or a large bed, into a room that is too small for it. It’s frustrating. Option 1: Move on to a bigger house. Option 2: Throw out…

data quality kpi

Data profiling is not data quality

For years, I have been a proponent of data quality measurement. Data quality cannot exist without management (and some would argue without governance). Meaningful data quality metrics play a critical role in managing data quality. Posts such as Don’t blow up the whale;   Changing data behaviour through KPIs, and  Accuracy, Completeness and Speed of Execution…

Big Data Quality for Hadoop

Big data quality

Ventana’s Research recent Big Data Integration benchmark survey  supports the growing awareness that data quality and integration are the principle time sinks for big data projects. There research finds that more than 50% of the time allocated to any big data project is taken in reviewing the data for quality and consistency – not surprising given…

Cosmos in the Free State

Data Profiling lessons from the Boer war

South Africa is a beautiful country. While international tourists may fly in to well known destinations, such as Cape Town or the Kruger National Park, locals tend to use the roads. A common sight at this time of the year are the swathes of wild Cosmos lining the road sides throughout much of the interior.…

Easter Egg Hunt

Find the hidden easter eggs in your data

The great annual, Easter Egg Hunt is over. On Sunday morning, millions of children, and quite a few adults, indulged in the ancient tradition of searching for hidden treasures to celebrate Easter. For techies, Easter Eggs are quirky features, such as jokes or games, that are intentionally hidden within other applications, for example, the flight…

data quality delusions

Are you suffering from data quality delusions?

Building a business case for data quality can be tricky. One of the biggest challenges – as data quality professionals we make assumptions about other people that hinder our ability to put together a compelling business case. How many of these assumptions do you make? Everyone understands how important data quality is Everyone wants to…

Business analysts critical to good data science

How sexy are your business analysts?

Last week I suggested that the role of data scientist may be best played by a team. Given the scarcity of these super humans, it makes far more sense to fill the role., if required, with a blend of roles. Many of the Data Science skills identified – statistician, SQL programmer, JAVA programmer etc –…