Shockingly, despite considerable investment and executive support, only about a third of big data projects are deemed successful. These were the findings of a Cap Gemini study cited in Information week, and echoed the same concerns that have been raised here over the last few years.

Scarcity of Skills

One of the primary challenges faced is the heavy reliance on scarce and expensive IT-oriented resources, such as data scientists, in open-source big data projects. These valuable resources end up spending nearly 60% of their time on data preparation tasks before they can delve into valuable analytics. An ABI Research report highlights the disproportional spending on services, whether through consultants or internal staff, which needs to be curtailed in big data projects.

Disconnect between Business and Data Analytics teams

Moreover, the time taken for data preparation creates a disconnect between business stakeholders, who possess domain knowledge, and big data technologists, who bring technical expertise. However, this issue can be addressed with big data discovery platforms that reduce the need for scarce technical skills, leading to a significant reduction in data preparation times.

The Complexity of Big Data platforms

Another stumbling block arises with the misconception that big data is synonymous with Hadoop. Unfortunately, off-the-shelf Hadoop platforms lack the enterprise-level traceability, lineage, and governance necessary for efficient data analysis. Consequently, companies struggle to find the data they need and data scientists often resort to shortcuts, leaving valuable data and money on the table.

Three Steps to Big Data Success

To achieve big data success, organizations must consider the following steps:

1. Build a Balanced Team: Staff the project with a blend of both business and IT experts. Technologies that cater to both technical and non-technical team members should be explored to maintain consistency in the application.
2. Identify the Right Use Case: Understand that big data is not a one-size-fits-all solution for all analytical problems. Instead, successful projects focus on specific departments, such as marketing, risk, or operations. Big data can rapidly provide insights into “what if” scenarios and reveal previously unknown patterns. Initial deployments may leverage the cloud to reduce the skills burden but can later be brought in-house.
3. Emphasize Big Data Governance: Debunk the myths that big data projects can be a free-for-all experimentation by data scientists. In reality, factors like access management, data lineage, business ownership, and accountability play a critical role in big data success.
4. Augment Hadoop with Additional Technologies: Recognize that “Hadoop alone cannot make a Big Data project successful.” Consider integrating add-on technologies, like Data360, which address challenges by enhancing governance and accelerating time to insight.

Download the “A Data Integrator’s Guide to Successful Big Data Projects” to delve deeper into the essential factors for achieving big data success.

By heeding these strategies and implementing data quality measures, businesses can navigate away from the big data trap and unlock the true potential of their enterprise information asset.