Managing the modern data pipeline

A few weeks back I wrote about the emerging role of the data engineer – the group of person’s responsible for delivering the quality data pipelines that enable the data scientist. I followed it up with this tweet – which I believe summaries very consisely the changing reality of big data and advanced analytics 2012…

true cost of ELT

Big data use case: Offloading the data warehouse to Hadoop

The true cost of ELT Today’s business world is demanding more from the data warehouse, because more than ever an organisation’s survival depends on its ability to transform data into actionable insights. However, ELT data integration workloads are now consuming up to 80% of database capacity, resulting in: Rising infrastructure costs Increasing batch windows Longer development cycles Slower…

Is data engineering overtaking data science

When Harvard Business Review first touted the data scientist as sexiest job of the 21st century  back in 2012 the role was still in its infancy. The promise of advanced analytics and the insights that business could gain – about their customers, their interactions, their products and everything else were rightly identified as potential gold. Data…