Big Data Headaches and 2016 Predictions

big data headachesBig Data?

Love it or hate it, the last year has shown that we cannot ignore it.

Most research shows that only around 25% of companies have deployed big data projects into production. This number is set to more than double over the next year as initial business cases show value, easier to implement technologies emerge and skills are developed.

With this in mind, what is the state of big data and where will it go this year?

  1. What is the role of big data in today’s age?

Simply put, big data uses new technologies and approaches to answer questions that could not be answered using traditional BI.

What kind of customer is most likely to buy our new product? What are the signals that a customer is about to churn? If we increase the number of times that we contact a customer in x period will that increase or decrease conversions?

The big data conversation should be focused on time to insight – how quickly can we get answers to critical business questions.

  1. What are two main advantages of using Big Data in companies ?

Historically BI has asked a limited set of questions against a (relatively) small subset of corporate data. Big data allows us to ask (almost) any question and test our hypotheses against a much more inclusive data set. So called dark data, the 80% of corporate data that is not currently analysed, can give valuable insights across diverse applications – including marketing and sales effectiveness, new product development, and even infrastructure optimisation and risk management.

Secondly, BI has often struggled to deliver within business timelines. Using appropriate big data platforms IT departments can provide managers with platform to support answers when they need them – within days or weeks instead of months or years

  1. How does Big Data drive companies and their CIO’s to take business Intelligence to the next level?

Where BI has historically been backwards looking (reporting on past events), big data applications are most commonly forward looking – analytics to test hypotheses. Big data should not be seen as a replacement for BI – instead, big data may be used to identify data segments for further BI reporting, to ask and answer throw away questions, and to better predict outcomes of alternative strategies.

Big data can also be used to optimise existing data warehousing environments by reducing ETL complexities and providing a searchable archive for older data.

  1. Why is the cost of acquiring, storing and managing data falling? (Please mention two main factors which contribute to this).

It’s not so much that costs are falling, as that costs are shifting. High end EDW type servers and applications have a relatively high cost per Terabyte.

 Platforms such as Hadoop provide high availability and high performance using infrastructure that is substantially lower cost.

 With open source big data, however, we have created a dependency on much more expensive resources – data scientists and related big data skilled professional are in short supply and high demand.

The real costs savings come when we can combine low cost storage with low cost resources. This typically means looking for tools that can hide the technical complexities of Hadoop and other low cost platforms, give you a future proof architecture, and provide governance, lineage, access control and other enterprise management capabilities.

The business case for big data is not cost savings

The reality is that, if you can save a few million a year on infrastructure and skills you will be able to store and analyse more data.

In most cases the business case for big data not about cost savings, but about the insights that you were previously unable to achieve. For example, a credit card vendor was able to infer a link between high value customers and the television channels they watch (interests). By targeting their ads at these interests they increased conversions dramatically with a substantially reduced budget. The insight saved tens of millions and delivered a much better result than

  1. What is the difference between reporting and analytics?

Reporting is backward looking. Did we achieve our sales targets? What percentage of our customers interacted with our call centre? Reporting is necessary to monitor whether our business has performed as expected.

Analytics, on the other hand, is about testing assumptions and answering what if questions.

“What if we opened our call centre for an extra two hours a day? Would it increase our sales revenue from our most profitable customer?”

“What is we preceded and followed up on a direct mail campaign with a telephone call. Would this increase or decrease conversions?”

“What signals do our customer’s give us before they churn? Can we use these signals to predict churn and take action to keep the client”

Big data allows us to consolidate and question diverse and complex data sets to give us insights that are not obvious from the enterprise data warehouse, or operational reports.

  1. How can CIO’s benefit by shifting their focus from reporting to analytics.

 Many CIO’s struggle to be seen as relevant to the business. Yes, it is critical that the email is working and that the network is up – but no one gets real business credit for keeping the lights on.

 In many organisations, reports are seen as part of that infrastructure. Many operational reports are outdated or irrelevant – KPIs and business priorities shift and BI struggles to keep up.

Analytics allows the CIO to have a direct impact on strategic decisions  by providing business with the answers they need to test hypotheses and optimise strategies.

Of course, this assumes that IT owns analytics.

Trends locally and overseas show that departmental big data deployments – for customer analytics, campaign management, risk management and manay more – are gaining ground as business managers get tired of waiting for an enterprise analytics strategy.

In my opinion, CIO’s still have the opportunity to deliver this value in Africa. They need to fil this gap before another exec does.

  1. Please elaborate on the  type of shift and the impact of this shift

Analytics has, largely, been restricted to whiz kids with PhDs in Statistics, or Applied Mathematics. These resources will always be in demand to build complex risk and other predictive models.

What big data is bringing, however, is the democratisation of analytics. It is allowing less technical business managers to test their gut feel before committing to a particular action – and to do it in business time without a dependency on the data scientist.

Big data also saves the data scientist time.

Surveys show that integrating and preparing data for analysis absorbs between 60% and 80% of the average analyst’s time. Analysts must spend time defining statistically relevant samples. Big data does not depend on samples – we test our hypothesis on all the data – and big data discovery tools cut integration and preparation time dramatically, The data scientist spends less time preparing data to answer simple questions and can deliver more of the advanced analyses that fit the job description.


  1. What considerations should be taken when implementing analytics with Big Data and reporting?

Henry Ford is famous for having said that, if he had asked his customers what they wanted, they would have asked for a faster horse.

 Companies need to ask whether existing BI vendors are simply providing a faster horse, or whether they are bringing something different.

Questions to ask should include:

  • Does the approach being chosen remove the data integration nightmare. Big data should not require structured schemas and SQL code in order to work.
  • Does the approach being chosen require hard to find skills, or long development cycles? R, Python, JAVA add complexity, time and cost, even if you can find skills.
  • Does the approach being taken address common data governance challenges. Where does my data come from, is the quality good enough, who should have access to it, how did we calculate this value? The management tools to support big data should be deployed from day 1.
  • Will the solution scale? Can I start in the cloud and the bring in house? Can I add additional data sources (internal and external)? Can I shift data from Hadoop to my EDW and back?
  • Is the architecture future proof? Will my analyses still work, and will it be optimised, as new frameworks emerge?
  • Do we want to democratise analytics? Can we enable business users to answer simpler, ad hoc questions and allow our data scientists and actuaries to focus their time on the complex stuff? How will we do this without losing control?
  • Does self service mean dash boarding or should it cover the entire analytics journey? We should enable a business user to integrate, aggregate and analyse data – not just pick a chart – if we really want to enable them to answer their questions timeously
  1. Please add any further interesting information which you would like to share

The most common application for big data is customer analytics – with over 60% of deployments having this focus. Financial services are big adopters of big data, but so are retail, telecommunications and government.

Some analysts are predicting that the Chief marketing officer will have a bigger IT budget that the CIO within the next few years – purely to meet the demand for better customer analytics. In leading organisations, however, the CIO and CMO are working closely together to align business and IT.

Big data is causing a realignment in corporate politics, not just in technology

This post is based on an interview with Itweb

One thought on “Big Data Headaches and 2016 Predictions

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.