MLOps engineering: A meta-synthesis of tools, practices and architectures for machine learning automation
DOI:
https://doi.org/10.31558/2786-9482.2024.2.3Keywords:
MLOps, automation, tools, frameworks, architecture, model deployment, ML-pipelines, meta-synthesisAbstract
Automating the end-to-end lifecycle of machine learning models is critical for their effective operationalization in production environments. Various tools, frameworks and architectures have emerged to support Machine Learning Operations (MLOps) practices. This paper presents a meta-synthesis of existing reviews to provide a comprehensive overview of enabling technologies for MLOps. The capabilities and features offered by popular commercial and open-source MLOps platforms are compared. Patterns in MLOps architecture and design philosophies are identified. The paper examines the role of containerization, orchestration, configuration management, and infrastructure automation in ML-pipelines. Approaches for model deployment on cloud and edge are also discussed. The following main results are obtained: 1) a meta-synthesis of systematic reviews was conducted to summarize knowledge about MLOps practices; it was determined that MLOps is a promising approach for effective deployment of machine learning models that requires further research; 2) relationships between MLOps principles, processes, and practices were analyzed. A diagram of the interconnections between key principles, stages of model development and implementation, and main MLOps practices is proposed; 3) the most effective MLOps practices for model deployment were identified – continuous integration/delivery, model and data versioning, ML pipeline automation, performance monitoring, experiment management, lifecycle management, data security and privacy, model explainability, data quality management, configuration management, deployment strategies, infrastructure automation, collaboration, risk management.
The results obtained have theoretical significance in generalizing and systematizing knowledge about MLOps practices and practical significance for implementing and improving MLOps processes in organizations.
References
Zahorodko, P. V., Semerikov, S. O., Soloviev, V. N., Striuk, A. M., Striuk, M. I., & Shalatska, H. M. (2021). Comparisons of performance between quantum-enhanced and classical machine learning algorithms on the IBM Quantum Experience. Journal of Physics: Conference Series, 1840, 012021. https://doi.org/10.1088/1742-6596/1840/1/012021
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023). Machine learning operations (MLOps): overview, definition, and architecture. IEEE Access, 11, 31866–31879. https://doi.org/10.1109/ACCESS.2023.3262138
Symeonidis, G., Nerantzis, E., Kazakis, A., & Papakostas, G. A. (2022). MLOps – definitions, tools and challenges. In 2022 IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC) (pp. 0453–0460). https://doi.org/10.1109/CCWC54503.2022.9720902
Testi, M., Ballabio, M., Frontoni, E., Iannello, G., Moccia, S., Soda, P., & Vessio, G. (2022). MLOps: A taxonomy and a methodology. IEEE Access, 10, 63606–63618. https://doi.org/10.1109/ACCESS.2022.3181730
Diaz-de Arcaya, J., Torre-Bastida, A. I., Zárate, G., Miñón, R., & Almeida, A. (2023). A joint study of the challenges, opportunities, and roadmap of MLOps and AIOps: A systematic survey. ACM Computing Surveys, 56, 84. https://doi.org/10.1145/3625289
Recupito, G., Pecorelli, F., Catolino, G., Moreschini, S., Nucci, D. D., Palomba, F., & Tamburri, D. A. (2022). A multivocal literature review of mlops tools and features. In 2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (pp. 84–91). https://doi.org/10.1109/SEAA56994.2022.00021
Steidl, M., Felderer, M., & Ramler, R. (2023). The pipeline for the continuous development of artificial intelligence models – current state of research and practice. Journal of Systems and Software, 199, 111615. https://doi.org/10.1016/j.jss.2023.111615
Lima, A., Monteiro, L., & Furtado, A. P. (2022). MLOps: practices, maturity models, roles, tools, and challenges – a systematic literature review. In Proceedings of the 24th International Conference on Enterprise Information Systems – Volume 1: ICEIS (pp. 308–320). https://doi.org/10.5220/0010997300003179
Haller, K. (2022). Managing AI in the enterprise: Succeeding with AI projects and MLOps to build sustainable AI organizations. Apress Berkeley, CA. https://doi.org/10.1007/978-1-4842-7824-6
e Oliveira, E., Rodrigues, M., Pereira, J. P., Lopes, A. M., Mestric, I. I., & Bjelogrlic, S. (2024). Unlabeled learning algorithms and operations: overview and future trends in defense sector. Artificial Intelligence Review, 57, 66. https://doi.org/10.1007/s10462-023-10692-0
Kolltveit, A. B., & Li, J. (2023). Operationalizing machine learning models: A systematic literature review. In Proceedings of the 1st Workshop on Software Engineering for Responsible AI (pp. 1–8). https://doi.org/10.1145/3526073.3527584
Calefato, F., Lanubile, L., & Quaranta, L. (2022). A preliminary investigation of MLOps practices in GitHub. In Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (pp. 283–288). https://doi.org/10.1145/3544902.3546636
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., … Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
Haertel, C., Staegemann, D., Daase, C., Pohl, M., Nahhas, A., & Turowski, K. (2023). MLOps in data science projects: A review. In 2023 IEEE International Conference on Big Data (BigData) (pp. 2396–2404). https://doi.org/10.1109/BigData59044.2023.10386139
Cohen, R. (2023). Digital strategy, machine learning, and industry survey of MLOps. In Digital Strategies and Organizational Transformation (pp. 137–150). https://doi.org/10.1142/9789811271984_0008
Sipe, T. A., & Curlette, W. L. (1996). A meta-synthesis of factors related to educational achievement: A methodological approach to summarizing and synthesizing meta-analyses. International Journal of Educational Research, 25, 583–698. https://doi.org/10.1016/S0883-0355(96)80001-2
Chrastina, J. (2018). Meta-synthesis of qualitative studies: Background, methodology and applications. In NORDSCI Conference proceedings (Vol. 1). https://doi.org/10.32008/nordsci2018/b1/v1/13
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019). Software engineering for machine learning: A case study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (pp. 291–300). https://doi.org/10.1109/ICSE-SEIP.2019.00042
Dhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021). Who needs to know what, when?: Broadening the explainable AI (XAI) design space by looking at explanations across the AI lifecycle. In Proceedings of the 2021 ACM Designing Interactive Systems Conference (pp. 1591–1602). https://doi.org/10.1145/3461778.3462131
Lwakatare, L. E., Crnkovic, I., & Bosch, J. (2020). DevOps for AI – challenges in development of AI-enabled applications. In 2020 International Conference on Software, Telecommunications and Computer Networks (SoftCOM) (pp. 1–6). https://doi.org/10.23919/SoftCOM50211.2020.9238323
Akkiraju, R., Sinha, V., Xu, A., Mahmud, J., Gundecha, P., Liu, Z., Liu, X., & Schumacher, J. (2020). Characterizing machine learning processes: A maturity framework. In D. Fahland, C. Ghidini, J. Becker, & M. Dumas (eds.), Business Process Management (Vol. 12168, pp. 17–31). Springer International Publishing. https://doi.org/10.1007/978-3-030-58666-9_2
Min, C., Mathur, A., Acer, U. G., Montanari, A., & Kawsar, F. (2023). SensiX++: Bringing MLOps and multi-tenant model serving to sensory edge devices. ACM Transactions on Embedded Computing Systems, 22, 98. https://doi.org/10.1145/3617507
Bachinger, F., Zenisek, J., & Affenzeller, M. (2024). Automated machine learning for industrial applications – challenges and opportunities. Procedia Computer Science, 232, 1701–1710. https://doi.org/10.1016/j.procs.2024.01.168
Filippou, K., Aifantis, G., Papakostas, G. A., & Tsekouras, G. E. (2023). Structure learning and hyperparameter optimization using an automated machine learning (AutoML) Pipeline. Information, 14, 232. https://doi.org/10.3390/info14040232
Bodor, A., Hnida, M., & Daoudi, N. (2023). Machine learning models monitoring in MLOps context: Metrics and tools. International Journal of Interactive Mobile Technologies (iJIM), 17, 125–139. https://doi.org/10.3991/ijim.v17i23.43479
Singh, P. (2023). Systematic review of data-centric approaches in artificial intelligence and machine learning. Data Science and Management, 6, 144–157. https://doi.org/10.1016/j.dsm.2023.06.001
Czakon, J., & Kluge, K. (2024). ML experiment tracking: What it is, why it matters, and how to implement it. https://neptune.ai/blog/ml-experiment-tracking
Peltonen, E., & Dias, S. (2023). LinkEdge: Open-sourced MLOps integration with IoT edge. In Proceedings of the 3rd Eclipse Security, AI, Architecture and Modelling Conference on Cloud to Edge Continuum (pp. 67–76). https://doi.org/10.1145/3624486.3624496
Melgar, L. A., Dao, D., Gan, S., Gürel, N. M., Hollenstein, N., Jiang, J., … Zhang, C. (2021). Ease.ML: A lifecycle management system for MLDev and MLOps. In 11th Conference on Innovative Data Systems Research, CIDR 2021. https://www.cidrdb.org/cidr2021/papers/cidr2021_paper26.pdf
Chen, H., & Babar, M. A. (2024). Security for machine learning-based software systems: A survey of threats, practices, and challenges. ACM Computing Surveys, 56, 151. https://doi.org/10.1145/3638531
Gopalakrishna, N. K., Anandayuvaraj, D., Detti, A., Bland, F. L., Rahaman, S., & Davis, J. C. (2023). “If security is required”: engineering and security practices for machine learning-based IoT devices. In Proceedings of the Fourth International Workshop on Software Engineering Research and Practice for the IoT (pp. 1–8). https://doi.org/10.1145/3528227.3528565
Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1–38. https://doi.org/10.1016/j.artint.2018.07.007
Rezazadeh, F., Chergui, H., Alonso, L., & Verikoukis, C. (2024). SliceOps: Explainable MLOps for streamlined automation-native 6G networks. IEEE Wireless Communications, 31, 224–230. https://doi.org/10.1109/MWC.007.2300144
Godwin, R. C., & Melvin, R. L. (2024). Toward efficient data science: A comprehensive MLOps template for collaborative code development and automation. SoftwareX, 26, 101723. https://doi.org/10.1016/j.softx.2024.101723
Yongqiang, D., Xin, W., Yongbo, L., & Wang, Y. (2020). Building network domain knowledge graph from heterogeneous YANG models. Journal of Computer Research and Development, 57, 699–708. https://doi.org/10.7544/issn1000-1239.2020.20190882
Neptune Labs. (2024). MLOps landscape in 2024: Top tools and platforms. https://neptune.ai/blog/mlops-tools-platforms-landscape
Gunny, A., Rankin, D., Harris, P., Katsavounidis, E., Marx, E., Saleem, M., Coughlin, M., & Benoit, W. (2022). A software ecosystem for deploying deep learning in gravitational wave physics. In Proceedings of the 12th Workshop on AI and Scientific Computing at Scale Using Flexible Computing Infrastructures (pp. 9–17). https://doi.org/10.1145/3526058.3535454
Vuppalapati, C., Ilapakurti, A., Chillara, K., Kedari, S., & Mamidi, V. (2020). Automating tiny ML intelligent sensors DevOPS using Microsoft Azure. In 2020 IEEE International Conference on Big Data (Big Data) (pp. 2375–2384). https://doi.org/10.1109/BigData50022.2020.9377755
Sothilingam, R., Pant, V., & Yu, E. S. K. (2022). Using i* to analyze collaboration challenges in MLOps project teams. In A. Maté, T. Li, & E. J. T. Gonçalves (eds.), Proceedings of the 15th International iStar Workshop (iStar 2022) co-located with 41th International Conference on Conceptual Modeling (ER 2022) (pp. 1–6). https://ceur-ws.org/Vol-3231/iStar22_paper_1.pdf
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
