On the Democratization of Machine Learning Pipelines

Alexandre Carqueja, Bruno Cabral, João Paulo Fernandes, Nuno Lourenço

Abstract: With the increase of Machine Learning (ML) adoption throughout many industries, the need for efficient methods to continuously design, develop, and deploy ML models has also grown. To address this issue, several ML pipelines have emerged with the goal of creating development environments which facilitate the deployment, evaluation and maintenance. In this paper, we advocate that most existing pipelines are not well suited for the initial stages of ML development, due to their high setup and maintenance overheads. As such, we propose a lightweight Quick Machine Learning framework, QML, which is capable of reducing the setup overhead and operating in the low infrastructure environments that are most common-place in experimental ML projects. To demonstrate QML's usefulness, we present a case-study where a lightweight ML pipeline was developed, and subsequently validated on a standard ML classification problem. Lastly, we assess the differences between our pipeline and an alternative lightweight workflow, based on DAGsHub. With this comparison, we conclude that our approach increases ML task automation as well as feature support, while falling short only in the Experiment Tracking category. To enable the broader community to experiment and assess QML, as well as the Lightweight Pipeline, this project has been made publicly available ¹ ¹ https://github.com/WALEX2000/qml1 https://github.com/WALEX2000/qml1 https://github.com/WALEX2000/qml1 https://github.com/WALEX2000/qml