In early April 2014, I was privileged to chair the Big Data and Analytics conference in Dubai. Big data is one of the arsenal of business intelligence tools available to modern businesses.

Business in Dubai finds itself in a similar position to South Africa: A little isolated from the mainstream in North America and Europe.

The lessons learned from the case studies and questions asked at this event help to make the business case for data quality clear.

Big Data – 3 Questions

In my introduction to the Dubai event, I asked three questions:

1.) Is big data just hype, or are there practical applications already?

Case studies from around the world showed that, while big data is in an early adoption phase, there are practical applications where it adds tremendous value. To my surprise, many of those featured used existing data better, rather than leaping into “new” data sources such as social media. At a practical level most featured Hadoop. Move forward nearly ten years and Hadoop is only one of a number of options for the data-lake, particularly as one looks at the Cloud.

2.) More data versus clean data. How important is data quality and data governance for big data success?

Overwhelmingly, the case studies supported my view that big data struggles with many of the data management challenges inherent in any analytics environment. Data quality remains a bugbear for data analysts and will continue to do so if it is not systemically addressed.

3.) What are the biggest hurdles to finding value?

A lack of skills coupled with the rapidly evolving Hadoop technology stack is some of the challenges mentioned. Privacy was a recurring theme, particularly with reference to big data in the cloud. Last but not least, was the challenge of finding appropriate big data use cases. Big data does not suit every requirement, nor is it a silver bullet to replace existing BI investments.

How have these insights aged?

Ten years later these insights still hold true. although the big data landscape has evolved beyond Hadoop. Hadoop skills are no longer the major hurdle – now the shortage is for Cloud engineers and the new low-cost storage options are technologies like Amazon S3.

When considering big data today you have more choices. You can stick with what you know – for example by moving to the Azure cloud or embracing new technologies from suppliers like Amazon or Snowflake

Before making a decision ask yourself the same three basic questions:

1. What is the practical application I want to implement?
2. Is cloud the right option or should I keep this on-premise?
3. Is the data of high integrity?

Poor quality data cannot deliver insights, no matter where you store it.

But remember, big data is not just about quantity: clean data versus more data. Quality reigns supreme in the quest for meaningful insights.

How will we govern the data? This may include considerations to add business context and make data easier to find, but may also need to consider governing access.

Transactional data quality isn’t just another cog in the machine; it’s the catalyst for efficiency.

Check out our guide to big data and business intelligence for more details on how to integrate big data into your BI capabilities

Legacy data siloes remain a barrier to business intelligence. Explore our guide to dealing with the challenges of implementing data analytics with legacy systems.