This post was first published by our partner, MANTA, on their blog, and is republished with permission.
The COVID-19 pandemic has introduced business challenges we have never had to deal with before. Experiencing the largest global recession since the Great Depression, companies across the globe are fighting for survival. Being over two months into the pandemic has taught us that a focus on efficiency is crucial today and that an increase in manual effort is one of the biggest data management hurdles in 2020. Performing everyday tasks manually, like you used to do before, feels like swimming against a powerful current. Not only does it affect overall productivity, but it’s also extremely challenging to perform well with everyone working remotely. According to Gartner, through 2022, manual data management tasks will be reduced by 45% through augmentation with smart metadata. Here’s how proper use of your metadata will help you come out of this situation on the winning side.
Metadata—You Are Doing it Wrong
It’s high time we perceived metadata as much more than just a set of data containing information about other data, such as its source, context, or movement patterns. It’s a good starting point, but your metadata has so much more power than its ability to tell you a “once upon a time” story about your data. With a proper tool, such as a unified lineage platform, metadata can turn into an active driver, augmenting your data management so you can automate repetitive tasks and shift the resources you currently have to more urgent tasks.
To better frame it, think about what it is like driving a car today compared to how it was in the past. Nowadays, thanks to augmented reality, our cars give us a heads-up when the windshield washer fluid should be topped up or the oil should be changed, and so on. Brakes even activate themselves if there is an object in front of the car! All this is possible because of a “metadata processor” in your car that scans the car itself, its parts, and then the environment and uses advanced intelligence to help you with various routine yet critical tasks.
Let’s get back to business. We have observed three major areas where automation is far superior to manual processes.
Let the Lineage Guide You
Why do organizations struggle to put metadata to good use and deprive themselves of the opportunity to quickly pivot their efforts during uncertain times? Typically, it is due to a lack of proper metadata and only using it for specific tasks. This makes it difficult or impossible to reuse it for other projects. Luckily, a proper automated lineage platform can help you overcome these obstacles. Here’s how it’s done.
- Lack of proper metadata
Most organizations know that their data is extremely valuable and that having it under control is the key to proper data management and making informed decisions. It is too bad that most of them only focus on current databases and structural metadata. Structural metadata shows how data sets are organized and sorted, but it does not provide information on how current data values are derived. The only way to get a full picture of the data environment and its true value is by implementing data lineage to show all the real sources and the changes made to the data you are working with before it reached its current state.
How can an organization deal with this situation when its data environment is huge, consisting of many technologies and platforms? Decoding lineage would simply require too much time, money, and resources. MANTA connects to various sources and transformation processes in your environment, gathers all the metadata, and reconstructs complete lineage. (You can find a complete list of supported technologies here.)
- Narrow scope of metadata use
Implementing a tool that uses metadata harvested from multiple technologies and applies it to a variety of tasks will save you significant time and resources. MANTA gives you a hand here too. Using MANTA Visualization, you can manage lineage at the level of detail that meets your users’ needs. They can search through all the lineage, focus only on selected parts, review previous versions, share lineage views with others in the organization, and a lot more. Let’s take a closer look at a few data management domains and what wonders augmentation can perform for them.
- Data warehousing, analytics, and DevOps. With automated data lineage in place, you can be sure that your teams will be able to provide continuous, high-quality software deliveries, even if a fully-remote mode is new to them. MANTA’s unified lineage allows your teams to understand even the most complex environments and to instantly see how planned changes will impact other parts of the environment. Having an accurate image of all data connections and dependencies makes it possible to perform instant root-cause analysis by tracking the cause of the issue back to its source. It also makes agile possible as it facilitates frequent releases by allowing engineers to automatically prepare even the smallest possible delivery package containing only the implemented changes and their dependencies. The same goes for sandboxing or testing just one part of your environment—integrating lineage with your source code repository, continuous integration, test management, and release management to speed up releases, enable intelligent sandboxing, increase testing efficiency, and improve the quality of final deliverables. All these capabilities augment your DevOps and agile practices, no matter how distributed your teams are.
- Data privacy and security. As long as everything goes smoothly, the privacy tools currently available seem to be enough—they provide automated PII discovery and data labeling, they support data access and deletion requests, and they come in handy when there’s a need to investigate a data breach. It gets a little wild when you need to run a sensitivity analysis or understand the sources of PII data to assess compliance with privacy policies. In these cases, privacy tools won’t help you understand how your data moves and how it’s linked across various locations where PII was discovered. A proper lineage tool gives invaluable assistance here—privacy specialists can immediately see data sources, how the data is connected, and how it moves, without reaching out to the IT team who have other priorities. Implementing a unified lineage platform takes the responsibility of PII tracking off the IT teams’ shoulders, enables self-service, and ensures data privacy and security by giving privacy specialists a tool they can actually work with without spending additional time and resources.
- Data quality. As I mentioned before, knowing how data values are derived is extremely important. Only by making sure that your data is solid, will you be able to make the decisions you should be making now—both quickly AND effectively. Lineage also plays a big role in quick and effective root cause analysis. Paired with your data quality tool that notifies you about data incidents, you can immediately start the investigation using MANTA, hunt down the incident source, and find other locations affected by the issue. With MANTA it takes just a few minutes, instead of several precious hours or days.
- Data governance. An automated approach allows organizations not only to speed up the process of delivering lineage but also to save resources, which is crucial, especially now, when the COVID-19 disruption is challenging organizations with increased absenteeism or team downsizing. A detailed picture of all data flows allows you to navigate your environment quickly, without spending additional time on asking and answering questions on complex parts of the environment. MANTA’s regular, automated scans also help you identify all changes and automatically update information in the data governance solution with no additional manual labor required.
One lesson learned from human history is that a lack of understanding and trust causes huge inefficiencies. It pops up in many different ways—data engineers can’t find data they need, data analysts hesitate to use data because they do not trust its quality or sources, or managers question reports that should support their daily decision making. The list of such examples is endless. But it doesn’t have to be this way. A properly managed unified lineage platform allows you to:
- automate many routine and manual tasks across data management
- adjust lineage to the needs of both technical and business professionals
- increase trust in data
- speed up collaboration of technology with governance, privacy, quality, and other teams, even if your people are not currently sitting in the same office
Think of staying productive during the pandemic as driving on an icy road for the first time. You are driving an old-school car without an anti-lock braking system (ABS). It starts to skid, and you are trying to control it. It’s not impossible, but the likelihood of an accident decreases when your vehicle is equipped with ABS and you can just hit the brakes, instead of figuring out the braking pressure.
Working in the pandemic puts our productivity to a test that none of us have studied for. The true power of augmentation is revealed when the human factor and leveraging known patterns both cooperate. Advance your data management with automated lineage that accelerates time-to-market, saves money, and uses the potential of the human brain to fully blossom and respond to unusual situations like the one we find ourselves in.
Has the COVID-19 disruption introduced new data management challenges for your organization?
Watch our webinar where MANTA’s CEO Tomas Kratky addresses current challenges.