Doing the same thing over and over again, and expecting a different result.

Albert Einstein

Think data in most businesses and the conversation invariably turns to analytics.

Indeed, business intelligence and analytics arguably get the lion’s share of data management spending. Yet, for all the investment very few organisations truly rely on their analytics teams to drive decision making.

Yet, we keep repeating the same mistakes, without addressing the challenges.

Culture eats strategy for breakfast

Peter Drucker

Creating a data-driven culture

One of the ongoing challenges is to create a data-driven culture – one where every decision maker from executives down to frontline knowledge workers trust analytics to guide them.

At one level this certainly requires an investment in the right tools. But more broadly, we need to give decision-makers the skills that they need to apply the insights that tools can provide, and we need to give them the confidence that the data (and insights derived) can be trusted.

Just as literacy involves the ability to read the written word and comprehend what you have read, data literacy involves the ability to source, interpret, and communicate data in context. These critical skills enable an organization to use data effectively for business actions and outcomes.

We expect information to be a business differentiator. Yet, in practice, only 25% of workers feel they use data effectively in their jobs, and only 21% of workers feel confident in their overall data literacy skills. It doesn’t matter how much data your business collects if your staff and decision-makers are not equipped to use it effectively.

Data literacy is a differentiator between companies that thrive and those that struggle to survive. Recent studies find that organizations with aggressive data literacy programs outperform those who have not prioritized data literacy. Data is viewed by many as the most valuable commodity of the 21st century. Companies that fail to harness their data will quickly lose relevance, reputation, and revenue.

Data literacy also reduces frustration by equipping business stakeholders to contribute to agile DataOps teams which, in turn, means that their needs are met more quickly.

DataOps is a collaborative practice that brings together diverse teams to streamline and automate the delivery of data across an enterprise.

It leverages data management technology and practices to ensure:

• Data Integration: simplifying the process of connecting to and consolidating disparate data sources
• Data Integrity: testing and improving data to ensure that business decisions are supported by accurate, complete data
• Metadata Management: maintaining and communicating a clear picture of the data estate, origin, dependencies and changes over time

This sound foundation engages both business and IT stakeholders in an agile way to deliver high quality, well-understood data to support analytics on demand.

Data literacy also helps business users to understand why they need to invest in a sound foundation of trusted data.

Data integrity remains a key challenge

The various technical and cultural challenges that inhibit the success of BI are not new.

In fact, as data volumes and complexity increase, so do the technical challenges facing BI teams, including data siloes, poor data quality, and data security.

Data integrity is the quality, reliability, trustworthiness, and completeness of a data set – providing accuracy, consistency and context.

Data integrity is built on four key pillars: enterprise-wide integration, accuracy and quality, location intelligence, and data enrichment.

Data integration

Most organisations must pull data together from a variety of databases and sources – both internal and, increasingly, external. While more agile technologies including data lakes and virtualisation engines are increasingly supplementing traditional data warehouses, these tools still require a combination of technical skills and data understanding.

“That limits scalability and increases the time it takes to analyze data,” said Ramesh Hariharan, CTO at consultancy LatentView Analytics.

He recommends creating a data catalogue that increases users’ data understanding by providing context on data sets.

Organisations must also grapple with the inconsistencies created through siloed systems. A lack of internal data standards across business units and departments can create huge challenges for BI teams looking to aggregate and consolidate data for enterprise reporting.

These inconsistencies can lead to multiple versions of the truth. Business users then see different results for KPIs and other business metrics that are labelled similarly in separate systems. To avoid that, companies must ensure clear definitions, policies and standards for data and understand data lineage.

Data quality

Analytics results can only be as accurate as the accuracy of the data that they are built on. Yet, in the rush to aggregate data for BI, many data quality issues are ignored.

The result frustrates both BI teams and decision-makers.

Decision-makers get recommendations that they can clearly see are inaccurate, or that leave them with a gut feeling that something isn’t right.

BI developers are left feeling unappreciated for the effort that they put in, or even, in the worst case, blamed for the poor quality of the report.

In the long run, management will typically reduce their investment in BI – after all, no one wants to pay for something that doesn’t add value.

A root cause is often the lack of understanding about the importance of proper data management among users. Data that is “good enough” for day to day operations may be woefully inadequate for reporting.

Organisations need a data strategy that considers the reporting needs necessary to support desired business outcomes, and builds, and maintains, the sound data foundation to deliver these needs.

Data Security

The final technical challenge that must be overcome is that of access management, particularly when analysing sensitive or personal data.

The intense focus on data privacy, driven by regulations such as Europe’s GDPR and South Africa’s PoPIA mean that BI teams have to ensure that access to data is limited to those that need it based on their legitimate business purpose. From an analytics perspective, fine grained access control policies must be defined and applied to analytics datasets – both on-premise and in the cloud.

This presents a unique set of challenges, particularly if data is still to be useful for analytics.

Teams must be able to apply anonymisation techniques – such as tokenisation or masking – in such as way as to preserve underlying attributes of the data for analysis. They must be able to set policies – to protect certain sensitive attributes from view for example – and apply these consistently across multiple data sets.

Certain users may also be restricted from entire rows of data – for example only having access to South African customers, or not being able to view data associated with children.

All of this must be done in such a way as to minimise performance impacts and maintenance overheads and must be done in such a way as to easily allow the Information Officer to provide meaningful audit reports to regulators and data subjects.

Trust is critical

Business intelligence and analytics investments will continue to suffer from poor adoption until we, as BI professionals, are able to shift business to rely on our data-driven assessments to drive decision making.

This, in turn, requires trust.

Change won’t happen in a day, but we all need to begin having these conversations with decision-makers, if we want to see improvement.