Big Data?

Love it or hate it, the last year has shown that we cannot ignore it.

We cannot ignore Big Data

Most research shows that only around 25% of companies have deployed big data projects into production.

This number is set to more than double over the next year as initial business cases show value, easier-to-implement technologies emerge and skills are developed

With this in mind, what is the state of big data and where will it go this year?

What is the role of big data in today’s age?

Simply put, big data uses new technologies and approaches to answer questions that could not be answered using traditional BI.

• What kind of customer is most likely to buy our new product?
• What are the signals that a customer is about to churn?
• If we increase the number of times that we contact a customer in x period will that increase or decrease conversions?

The big data conversation should be focused on time to insight – how quickly can we get answers to critical business questions.

What are the two main advantages of using Big Data in companies ?

Historically, BI has asked a limited set of questions against a (relatively) small subset of corporate data.

Big data allows us to ask (almost) any question and test our hypotheses against a much more inclusive data set.

Access to dark data

So-called dark data, the 80% of corporate data that is not currently analysed, can give valuable insights across diverse applications – including marketing and sales effectiveness, new product development, and even infrastructure optimisation and risk management.

Quicker delivery

Secondly, BI has often struggled to deliver within business timelines.

Using appropriate big data platforms IT departments can provide managers with a platform to support answers when they need them – within days or weeks instead of months or years

How does Big Data drive companies and their CIO’s to take business Intelligence to the next level?

Where BI has historically been backwards looking (reporting on past events), big data applications are most commonly forward-looking – analytics to test hypotheses.

Big data should not be seen as a replacement for BI – instead, big data may be used to identify data segments for further BI reporting, to ask and answer throw-away questions, and to better predict outcomes of alternative strategies.

Big data can also be used to optimise existing data warehousing environments by reducing ETL complexities and providing a searchable archive for older data.

Why is the cost of acquiring, storing and managing data falling?

It’s not so much that costs are falling, as that costs are shifting.

High-end EDW-type servers and applications have a relatively high cost per Terabyte.

 Platforms such as Hadoop provide high availability and high performance using infrastructure that is substantially lower cost. Modern Cloud storage options like Amazon S3 offer low-cost data archiving. These, and similar technologies, offer the ability to store and query more data at lower costs.

 With open-source big data, however, we have created a dependency on much more expensive resources – data scientists and related big data skilled professionals are in short supply and in high demand.

The real cost savings come when we can combine low-cost storage with low-cost resources.

This typically means looking for tools that can hide the technical complexities of Hadoop and other low-cost platforms, give you a future-proof architecture, and provide governance, lineage, access control and other enterprise management capabilities.

The business case for big data is not cost savings

The reality is that, if you can save a few million a year on infrastructure and skills you will be able to store and analyse more data.

In most cases the business case for big data is not about cost savings, but about the insights that you were previously unable to achieve.

For example, a credit card vendor was able to infer a link between high-value customers and the television channels they watch (interests). By targeting their ads at these interests they increased conversions dramatically with a substantially reduced budget. The insight saved tens of millions and delivered a much better result than traditional BI could have.

What is the difference between reporting and analytics?

Reporting is backwards looking.

• Did we achieve our sales targets?
• What percentage of our customers interacted with our call centre?

Reporting is necessary to monitor whether our business has performed as expected.

Analytics, on the other hand, is about testing assumptions and answering what-if questions.

• What if we opened our call centre for an extra two hours a day? Would it increase our sales revenue from our most profitable customer?
• What if we preceded and followed up on a direct mail campaign with a telephone call. Would this increase or decrease conversions?
• What signals do our customers give us before they churn? Can we use these signals to predict churn and take action to keep the client?

Big data allows us to consolidate and question diverse and complex data sets to give us insights that are not obvious from the enterprise data warehouse, or operational reports.

How can CIO’s benefit by shifting their focus from reporting to analytics.

Many CIOs struggle to be seen as relevant to the business.

Yes, it is critical that the email is working and that the network is up – but no one gets real business credit for keeping the lights on.

In many organisations, reports are seen as part of that infrastructure.

Many operational reports are outdated or irrelevant – KPIs and business priorities shift and BI struggles to keep up.

Analytics allows the CIO to have a direct impact on strategic decisions by providing business stakeholders with the answers they need to test hypotheses and optimise strategies.

Of course, this assumes that IT owns analytics.

Trends locally and overseas show that departmental big data deployments – for customer analytics, campaign management, risk management and many more – are gaining ground as business managers get tired of waiting for an enterprise analytics strategy.

In my opinion, CIOs still have the opportunity to deliver this value in Africa. They need to fill this gap before another exec does.

Please elaborate on the  type of shift and the impact of this shift

Advanced Analytics has, largely, been restricted to whiz kids with PhDs in Statistics, or Applied Mathematics.

These resources will always be in demand to build complex risk and other predictive models.

What big data is bringing, however, is the democratisation of analytics.

It is allowing less technical business managers to test their gut feeling before committing to a particular action – and to do it in business time without a dependency on the data scientist.

Big data also saves the data scientist time.

Surveys show that integrating and preparing data for analysis absorbs between 60% and 80% of the average analyst’s time.

Analysts must spend time defining statistically relevant samples.

Big data does not depend on samples – we test our hypothesis on all the data – and big data discovery tools cut integration and preparation time dramatically.

The data scientist that spends less time preparing data to answer simple questions can deliver more advanced analyses that fit the job description.

What considerations should be taken when implementing analytics with Big Data and reporting?

Henry Ford is famous for having said that, if he had asked his customers what they wanted, they would have asked for a faster horse.

 Companies need to ask whether existing BI vendors are simply providing a faster horse, or whether they are bringing something different.

Questions to ask BI vendors should include:

• Does the approach being chosen remove the data integration nightmare. Big data should not require structured schemas and SQL code in order to work.
• Does the approach being chosen require hard-to-find skills or long development cycles? R, Python, JAVA add complexity, time and cost, even if you can find skills.
• Does the approach being taken address common data governance challenges? Where does my data come from, is the quality good enough, who should have access to it, and how did we calculate this value? The management tools to support big data should be deployed from day 1.
• Will the solution scale? Can I start in the cloud and then bring on-premise, or vica versa? Can I add additional data sources (internal and external)? Can I shift data from Hadoop or S3 buckets to my EDW and back?
• Is the architecture future-proof? Will my analyses still work, and will they be optimised, as new frameworks emerge?
• Do we want to democratise analytics? Can we enable business users to answer simpler, ad hoc questions and allow our data scientists and actuaries to focus their time on the complex stuff? How will we do this without losing control?
• Does self-service mean dashboarding or should it cover the entire analytics journey? We should enable a business user to integrate, aggregate and analyse data – not just pick a chart – if we really want to enable them to answer their questions timeously

Please add any further interesting information which you would like to share

The most common application for big data is customer analytics – with over 60% of deployments having this focus. Financial services are big adopters of big data, but so are retail, telecommunications and government.

Some analysts are predicting that the Chief Marketing Officer will have a bigger IT budget than the CIO within the next few years – purely to meet the demand for better customer analytics. In leading organisations, however, the CIO and CMO are working closely together to align business and IT.

Big data is causing a realignment in corporate politics, not just in technology

This post is based on an interview with Itweb