The key to understanding what data engineering lies in the “engineering” part. Engineers design and build things. “Data” engineers design and build pipelines that transform and transport data into a format wherein, by the time it reaches the Data Scientists or other end users, it is in a highly usable state. These pipelines must take data from many disparate sources and collect them into a single warehouse that represents the data uniformly as a single source of truth.
Sounds simple enough but a lot of data literacy skills goes into this role. This is why Data Engineers are in such short supply and why there is confusion around the role. The figure below is one example of the activities involved in data engineering.
Monica Rogati, an equity partner at Data Collective, created a now-famous data science hierarchy of needs. It depicts where data engineering falls in the roadmap to becoming a data science/AI-driven organization.
A Data Engineer’s role is at level 2 and 3. It’s worth noting that the bottom level “collect” is growing larger and larger, thereby driving the need for more Data Engineers.
(BIG) DATA ENGINEERS ARE IN DEMAND
Data Scientists as a professional group get a lot of attention and hype. Over the last several months, however, we’ve seen a growing interest in using our technical skills testing platform for data engineering roles.
We understand intuitively the surge in demand for Data Engineer skills testing. LinkedIn’s 2020 Emerging Jobs Report and Hired’s 2019 State of Software Engineers Report ranked Data Engineer jobs right up there with Data Scientist and Machine Learning Engineer.
However, for some companies, especially those still finding their legs in data science or AI, it’s not always apparent what data engineering is, what role Data Engineers play within the analytics team and what skills are required (and should be vetted) to do the job.
So we thought in this brief article we’d answer the question “what is data engineering?”. We also explain why it is now widely recognized as being extremely important and what the role and skillsets of a Data Engineer are. It’s important to note that the definition of what data engineering is and what a Data Engineer does continues to evolve, so consider this summary a “snapshot”.