This article reflects on my past decade’s experiences and observations regarding the data market’s evolution. Over the years, data automation has transformed traditional roles, merging the distinct sectors of analysts, engineers, and scientists into hybrid positions that require a broader skill set and a more generalised approach. A decade ago, technological advancements involved distinct specialisation in analytics, engineering, or data science. Today, roles like Full Stack Data Scientist and Analytics Engineer are emerging as leaders in the field, emphasising flexibility rather than restraint to specific niches.
In today’s business landscape, the significance of data cannot be overstated. An increasing volume of data streams through pipelines, and the importance of cloud technologies grows daily. Traditional sectors such as advertising and insurance are undergoing automation, transitioning from Excel spreadsheets to cloud-based solutions. While some data analysts may still present findings in Excel or PowerPoint due to the needs of non-technical audiences, this is more a reflection of the audience’s familiarity than a lack of cloud capabilities in those fields.
Conversely, industries like telecommunications, satellite communications, energy, and banking increasingly rely on Big Data skills, driven by the inflow of streaming machine data measured in zettabytes. In smaller data-focused fields, questions about data size sometimes feel trivial, but in my experience, the answer can be overwhelming: the data is virtually infinite, arriving every second. The reality is that data is everywhere, and its importance will only continue to grow.
My journey with data began while studying for my mathematics degree. I am grateful for the connections I made during that time, as they allowed me to work on significant problems such as the Ebola outbreak and copula modelling for behavioural data, deepening my fascination with statistics. I graduated as one of the top students in my statistics program, and earning my degree while working in data created a unique blend of knowledge. This foundation enabled me to write my thesis under the supervision of the statistics program director. Reflecting on my journey over the last ten years, I chose my degree and career path wisely.
2015: A Clear Separation in the Data World
In 2015, when I began attending my first data science conferences and participating in volunteer projects utilising my statistical knowledge—such as analysing data on the Ebola outbreak or studying students’ behaviour in research from UCL —the data landscape looked markedly different. There was a distinct separation between Data Engineer, Data Scientist, and Data Analyst roles. As a mathematics student, I found this division fascinating yet somewhat limiting, as I intuitively felt these areas were interconnected. I knew I wanted to work in data.
When my professors asked if I intended to pursue a career in data mining, I responded, “Well, no, I want to build models and make predictions rather than just visualise existing data.” I recognised that Machine Learning is fundamentally different from traditional statistics. Upon reflection, it’s clear that building models and making predictions is precisely what statisticians have done for decades. However, at the time, I was unaware of the evolution of data roles, and my professors asked their questions based on their experiences when they had not transitioned back to academia. Their focus was on theoretical knowledge, while my interest lay in the practical impact of that theory.
Now, we understand that Machine Learning is rooted in statistical theory. Various tools have emerged to handle the theoretical aspects, leading to the contraction of the Data Scientist job title and the emergence of new roles: the boundaries between Data Engineer, Data Scientist, and Data Analyst are blurred; I was drawn to all three roles and enjoyed aspects of each. The more ordinary tasks have since been automated, allowing us to explore a bit of everything.
Yet, back in 2015, there was a clear line between these three sectors, and organisations typically required three distinct individuals to complete the entire data cycle. This separation underscored the specialised nature of each role and highlighted the growing need for collaboration across these domains.
2018: Entering the Workforce and Emerging New Data Fields
I graduated in 2018 and landed my first fully paid job in data at a small marketing consultancy. Before this role, I encountered an episode related to DevOps data. I didn’t fully understand what DevOps meant then, as it was misrepresented to me as solely dealing with machine data. While there are overlaps, DevOps is fundamentally separate from the data field. Data professionals certainly benefit from DevOps principles like continuous improvement, and the hybrid evolution of data jobs can be traced back to the DevOps movement, but they are distinct domains.
In my first role at the marketing consultancy, I worked with models familiar from my university coursework, including linear regression, Bayesian statistics, and time series analysis. I quickly realised that companies were marketing this statistical knowledge—often derived from statistics and economics graduates—as Machine Learning models. During my studies, we forecasted using time series analysis, and although we didn’t exclusively use linear regression for forecasting, we understood that it could be applied in that context. This realisation highlighted the confusion surrounding the application of statistics for business purposes.
Since then, I have employed time series forecasting for the satellite launch, utilised clustering techniques for Quality of Experience on voice calls, and applied regression models for churn analysis. I learned to adapt my terminology to align with industry standards, referring to my work as machine learning, even though the methodologies mirrored what I had learned in my statistics coursework. However, this is not my most significant interest, as the statistical models can be automated.
At this time, I also witnessed the emergence of cloud technologies. I met many women eager to enter the cloud engineering field, learning various cloud technologies and striving to secure this booming job title. I often heard, “You need to start working with Cloud; it’s going to be a big thing.” However, working at a media agency, where data was relatively small and manageable, I struggled to grasp the necessity of cloud technologies, as everything fit neatly into Power BI dashboards or Excel spreadsheets.
Landing a job at Matchesfashion was a pivotal moment for me—it opened my eyes to the sheer scale of data and highlighted the importance of cloud knowledge. Although I observed emerging roles like Machine Learning Engineers and Cloud Architects, I initially thought they were reserved for those with a computer science background, and I doubted my eligibility for such positions. Nonetheless, I recognised that my background in statistics equipped me to navigate the data field successfully, and I embraced the opportunities before me.
2020-2021: The Shift in Industry Needs
The COVID-19 pandemic pushed the digital revolution to new heights, forcing businesses to adapt rapidly to online operations. With the overflow in demand for digital solutions, companies needed results yesterday. This environment created a boom in data contracts and a rich market, but it also accelerated the evolution of data science automation. After all, how many times can one build the same time series forecaster or regression model from scratch? Companies recognised the necessity of automation wherever possible.
Conversations with data leaders revealed a common theme: “We need someone who can jump in and quickly address whatever challenges arise.” The demand for Data Scientists who could build models from scratch diminished; instead, companies sought individuals who could leverage tools to automate machine learning processes. These professionals needed to understand the models well enough to debug issues when necessary and supervise algorithms to ensure data quality.
The need for skilled individuals who could clean data, select variables, visualise outputs, and validate results became essential, especially with the rise of automation tools like BigQuery ML. This period marked a strategic turning point where the role of Data Scientist began to merge more closely with that of Data Analyst, particularly for those equipped with machine learning knowledge. The emphasis shifted toward speeding the process from data cleaning to deriving valuable insights and predictions.
While Data Scientists increasingly focused on large language models (LLMs) and deep learning (DL), many machine learning models became automated. This evolution signified the emergence of a hybrid role that combined the responsibilities of a Data Analyst with a foundational understanding of data science. As a result, professionals in the field became more adaptable, adapting to the changing landscape and addressing the growing need for efficient data-driven decision-making.
2022 and Beyond: Discovering Analytics Engineering
In 2022, the data market faced significant challenges. As one recruiter I know remarked, “This is the worst data market I’ve seen in ages.” It was a period marked by quietness and numerous layoffs. The hiring boom that followed the post-COVID digital revolution led to a contraction in the data market, returning it to more average conditions. While this should not have been surprising to anyone familiar with economic cycles, it was nonetheless a disheartening experience for those affected by layoffs.
Fortunately, I was secure in my role at Inmarsat, working alongside a team of engineers in the telecommunications sector. The uniqueness of my position at Inmarsat became apparent as we navigated this evolving landscape. At the time, I was unsure if we knew the term “Analytics Engineer,” but I already embodied that role. We recognised the need for someone with a data analytics background who could bridge the gap between data engineers and the network intelligence team.
Tools like DBT were emerging as robust solutions, helping to restructure data from internal legacies to Google Cloud Platform (GCP). This transition brought new challenges, but the most critical requirement was having someone who understood data analytics well enough to clean, assess data accuracy, validate outputs, and more.
As 2022 progressed, it became clear that other companies also recognised the need for a hybrid role that combined the skills of both Analyst and Engineer. This realisation led to the formalisation of the title “Analytics Engineer.” By 2023, the first recruitment agency specialising in Analytics Engineering will emerge in the UK market, signalling a significant shift in how organisations seek to fill this crucial role. This evolution reflects the growing importance of blending analytical expertise with engineering skills to meet the demands of modern data-driven decision-making.
Conclusion
The title “Analytics Engineer” is a positive development, although it reflects some outdated thinking from when Analysts, Engineers, and Scientists were viewed as three distinct roles. Today, industries increasingly require hybrid professionals who can streamline the decision-making process. However, through discussions at various meetups, I’ve observed that some companies have misinterpreted this role. They envision the Analytics Engineer as merely a liaison between the Engineering and Analytics departments and see Analytics Engineering as a separate entity, which diverges from the true intent of the position.
The industry should focus on shortening the path from data acquisition to business decision-making rather than complicating it. This confusion is partly fueled by the marketing surrounding tools like DBT, which, while excellent, has led some to believe it is the only solution for Analytics Engineers.
As the market evolves, it will soon recognise that the push to merge data science with analytics—driven by the automation of machine learning models—parallels the integration of data engineering and analytics roles through the creation of the Analytics Engineer position. Organisations need individuals to facilitate and expedite the journey from data sources to actionable business insights. Professionals with extensive experience, like myself, who identify as Data Science Full Stack, will be essential in navigating this evolving landscape.
This is where the DevOps concepts come in place: continuous improvement driven by automation and technological advancements and adaptation to the hybrid positions created by this movement.
