ML-Enabled Systems Model Deployment and Monitoring: Status Quo and Problems
CoRR(2024)
摘要
[Context] Systems incorporating Machine Learning (ML) models, often called
ML-enabled systems, have become commonplace. However, empirical evidence on how
ML-enabled systems are engineered in practice is still limited, especially for
activities surrounding ML model dissemination. [Goal] We investigate
contemporary industrial practices and problems related to ML model
dissemination, focusing on the model deployment and the monitoring of ML life
cycle phases. [Method] We conducted an international survey to gather
practitioner insights on how ML-enabled systems are engineered. We gathered a
total of 188 complete responses from 25 countries. We analyze the status quo
and problems reported for the model deployment and monitoring phases. We
analyzed contemporary practices using bootstrapping with confidence intervals
and conducted qualitative analyses on the reported problems applying open and
axial coding procedures. [Results] Practitioners perceive the model deployment
and monitoring phases as relevant and difficult. With respect to model
deployment, models are typically deployed as separate services, with limited
adoption of MLOps principles. Reported problems include difficulties in
designing the architecture of the infrastructure for production deployment and
legacy application integration. Concerning model monitoring, many models in
production are not monitored. The main monitored aspects are inputs, outputs,
and decisions. Reported problems involve the absence of monitoring practices,
the need to create custom monitoring tools, and the selection of suitable
metrics. [Conclusion] Our results help provide a better understanding of the
adopted practices and problems in practice and support guiding ML deployment
and monitoring research in a problem-driven manner.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要