IML4E – Pipeline Probe

Pipeline Probe is a novel, open-source tool designed for automated quality assessment of machine learning (ML) models within MLOps pipelines. Its key features include seamless integration with popular MLOps pipelines like Kubeflow through a plugin mechanism, adaptability to various ML quality assessment tools, and the ability to work alongside existing tools and libraries without the need for extensive code changes.

In the rapidly growing field of machine learning, quality assessment is crucial to ensure the reliability and performance of ML models. However, the current landscape of quality assessment tools lacks customization, requiring substantial code adaptation for each use case. Pipeline Probe addresses these challenges by providing automated quality assessments through a seamless integration of various quality assessment tools and libraries without the need for extensive code changes. 

Pipeline Probe works by analysing artifacts produced during the MLOps cycle, such as data sets, model architecture, model parameters, performance metrics, evaluation results, feature importances, model explanations, and model robustness. It utilizes a plugin system to incorporate various ML quality checkers for evaluating each artifact with the appropriate measurement, providing a comprehensive and accurate analysis of both model and data quality. Plugins for Kubeflow with MLFlow tracking are included and thus, these tools are supported out of the box. Further, a data quality test suite that uses ISO 25012/24 standard to make quality assessments on data and a model robustness test application using Adversarial Robustness Toolbox (ART) are also included as companion applications to the Pipeline Probe. 

Figure 1 shows the architecture diagram of the Pipeline probe. As can be seen, the tool comprises two primary components. The first is the collection of plugins which collect, analyze, and evaluate artifacts throughout the life cycle. Each plugin is a separate program running on a particular port. Second key component is the core that registers, manages, and has information about status of all the plugins. The core and the plugins use gPRC calls to communicate.