Don’t get stuck in a big data trap!

An Information Management article repeats strategies for Solving the Big Data “Abandonment” problem

The article cites research by Cap Geminii claiming that, in spite of executive support and considerable spend, only around a third of big data projects are considered to have been successful. The reasons, and proposed solutions, are consistent with thise that I have been promoting here over the last few years.

One challenge – open source big data projects are dependent on scare (and expensive) IT oriented resources such as data scientists.

This has various impacts.

Expensive resources are spending almost 60% of their time in data preparation tasks that before they can get to the value adding analytics. An ABIresearch report suggests that the level spending on services (whether with consultants or internal staff) in big data projects is disproportional and must be reduced.

The data preparation tasks also increase the time that it takes for any big data project to deliver value, and create a disconnect between the business stakeholders, who have the domain knowledge, and the big data technologists who have the technical skills. By contrast, big data discovery platform reduce the need for scare technical skills, and reduce data preparation times dramatically.

Another challenge: While big data has become synonymous with Hadoop, off the shelf Hadoop platform do not provide enterprise strength trace-ability, lineage and governance.

In effect, companies are unable to find the data that they need to analyse. The complexities of managing, preparing, identifying and analysing big data force data scientists to take short cuts that leave data and money on the table.

The article repeats calls that I have made consistently on this blog over the last few years:

In order to achieve big data success companies must:

  1. Staff the project with a blend of business and IT staff. This also means investigating technology that supports both technical and non technical big data team members via a consistent application
  2. Identify the right big data use case. Big data is not the solution to all analytics problems. Successful projects have a departmental focus – e.g. marketing or risk or operations. Big data can be used to rapidly answer “what if” type questions and uncover previously unknown insights. Initial deployments may take advantage of the cloud to reduce the skills burden, but can be brought in house later,
  3. Big data governance is a key success factor. The myths that big data is about a team of data scientists experimenting on data with no specific aim, that data quality is unimportant, and that planning and documentation are unnecessary have been proven wrong. Factors such as access management, lineage, business ownership and accountability are critical.
  4. Finally the report finds that “Hadoop alone cannot make a Big Data project successful.” Corporates must look at add on technologies, like Datameer, that solve some of the challenges described above by adding governance and reducing time to insight.

Download the “Big Data Buyers Guide” to learn more about the factors to consider to achieve big data success.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.