A recent post by Alastair Adam on LinkedIn has brought to light a staggering statistic that might send shockwaves through the data science community. It’s a harsh reality check: data scientists, on average, change jobs every 19 months. To put this into perspective, that’s roughly a year and a half before they decide to move on. But here’s the kicker – during this entire period, they spend a whopping 12 months onboarding and settling in and only contribute a mere 30 days of actual value.

So, in essence, for 19 months of employment, you get a little over 30 days of productive output.

It’s a jaw-dropping revelation that raises critical questions about the expectations and the realities of data science careers.

The Onboarding Odyssey

Alastair Adam points out that a significant chunk of these 19 months is devoted to onboarding. This includes familiarizing oneself with the data landscape, building relationships with key stakeholders, and gaining a deep understanding of the available data resources. It’s an essential phase in any data scientist’s journey, but it takes a substantial toll on the time available for real work.

The Data Hunt

Niel Burge chimes in with another sobering fact: During the precious seven months when data scientists are supposed to be hitting their stride, a staggering 80% of their time is spent finding and preparing data. Only 20% of their time is left to engage in actual, meaningful work.

“And in those 7 months when they are hitting their stride we know they spend 60% of their time hunting data (Monday-Wednesday), 20% cleaning and organising data (Thursday) and only have 20% to do real work (Friday).”

Neil Burge

The numbers don’t lie. Data scientists are grappling with a continuous data hunt, a tiring data cleaning process, and a mere fraction of their week dedicated to applying their expertise to add value. This paints a stark picture of their job satisfaction and effectiveness.

The Expectation-Reality Gap

What’s causing this rapid churn? Alastair Adam posits that one of the primary culprits is the gaping chasm between what data scientists are promised and what they actually experience. Often, they are told they will be “changing the business” through groundbreaking AI and ML capabilities. The reality, however, is that much of their time is consumed by the laborious task of finding and preparing data.

It’s this stark disconnect between expectations and reality that breeds discontent among data scientists. They enter their roles with high hopes of being data superheroes, but the day-to-day grind tells a different story.

Investing in the Supporting Cast

While organizations are keen to invest in advanced analytics and data science skills and tools, very few are making adequate investments in the supporting capabilities – data governance, data quality, and metadata management – that provide the much-needed business context and reduce the frustrations experienced by data scientists.

When data preparation work is required, having the right tools can make all the difference. These tools not only speed up the process but also facilitate knowledge sharing and reuse. Moreover, they enable collaborative approaches like DataOps, which allow data scientists to receive support from data engineers and other stakeholders, thus increasing productive work hours.

A Leap Towards Productivity

Imagine if we could transition from just 30 days of productive time over 19 months to 60 or even 90 days. That’s a remarkable 200 to 300% improvement in value delivered. More importantly, by simplifying their daily tasks and increasing the amount of time data scientists spend on actual data science, perhaps we can keep them engaged and satisfied in their roles for a more extended period.

Of course, it’s essential to understand that technology isn’t a silver bullet. It requires discipline and strategic implementation. However, the right technology choices can undoubtedly be a significant part of the solution to bridge this glaring gap between expectations and reality.

In conclusion, the data science churn rate and the limited days of value added are serious concerns that organizations need to address. By investing in the right tools and support systems and by setting realistic expectations, we can unlock the full potential of data scientists and ensure that they don’t just pass through our doors but become valuable, long-term assets to our data-driven organizations. After all, in the world of data, quality always matters.

Connect with Alastair Adam on Linkedin