The results of Syncsort’s 2018 Big Data Trends survey have been releases and can be accessed as a ebook by following the link.
One clear result of the survey, which engaged several hundred IT professionals over multiple industries is that what was unfamiliar technology four years ago is now becoming an integral part of the IT landscape.
Over 40% of respondents have Hadoop or Spark in production, while another 30% are in some form of proof of concept.
The most common use cases revolve around the abilities of Hadoop and Spark to manipulate and integrate data. Data lakes are becoming common place and will increasingly replace traditional ETL.
RDMS data remains the most common source – but the variety of data supplementing this is ever increasing.
Most respondents see the data lake as an opportunity to increase business user productivity by making enterprise data more readily available, amongst numerous other benefits, including cutting costs, being more responsive, and enhancing decision making,
In order to deliver these benefits, however, respondents need to overcome numerous challenges.
Data governance and data quality emerge as biggest challenges to big data
Concerns over a shortage of big data skills, which has been the number one hurdle over the last few years, have now been overcome by the realities that poor governance and poor quality in the data lake are the biggest challenges facing big data implementations.
For executives and regulators to trust the insights delivered thorough big data analytics companies must implement data governance processes that allow them to easily find, understand and trust the data.
Answers to questions such as “where did it come from”, “who has access to it”, “how has it been manipulated”, and “what was the quality/confidence level” are becoming increasingly essential.
Data governance capabilities need to include the data lake but must also extend to the tradtional BI and operational data space.
Similarly, users must be able to assess and improve the quality of data in the data lake, as well as in traditional RDMS spaces.
To get all of the insights from this survey your should read the ebook