A Survey-Based Evaluation of the Data Engineering Maturity in Practice.

The proliferation of data-intensive applications is continuously growing. Yet, many of these applications remain experimental or insular as they face data challenges that are rooted in a lack of practical engineering practices. To address this shortcoming and fully leverage the data resource, a professionalization of engineering data-intensive applications is necessary. In a previous study, we developed a data engineering reference model (DERM) that outlines the important building-blocks for handling data along the data life cycle. To create the model, we conducted a systematic literature review on data life cycles to find commonalities between these models and derive an abstract meta-model. We validated DERM theoretically by classifying scientific data engineering topics on the model and placed them in their corresponding life cycle phase. This led to the realization that the phases plan, create and destroy, as well as the layers enterprise and metadata are underrepresented in research literature. To strengthen these findings, this work conducts an empirical survey among data engineering professionals to assess the maturity of the model’s pillars from a practical perspective. It turns out that the gaps found in theory also prevail in practice. Based on our results, we derived a set of research gaps that need further attention for establishing a practically grounded engineering process.