COPA: Comparing the incomparable in multi-objective model evaluation

Adrián Javaloy, Antonio Vergari, Isabel Valera

Keywords: multi-objective, model selection, multi-criteria evaluation, relative rankings

TL;DR: We investigate how objectives can be automatically normalized and aggregated to systematically navigate their Pareto front.

Abstract: As machine learning (ML) practitioners, we often have hundreds of (trained) ML models at hand from which we need to choose one, based on various objectives such as accuracy, robustness, fairness, scalability, etc. However, how to _compare_, _aggregate_ and, ultimately, _trade-off_ these objectives is usually a time-consuming task that requires of expert knowledge, as they may be measured in different units or scales. In this work, we investigate _how_ objectives can be automatically normalized and aggregated to systematically navigate their Pareto front. To do so, we make incomparable objectives comparable using their CDFs, approximated by their relative rankings. As a result, we can aggregate them while matching user-specific preferences, allowing practitioners to meaningfully navigate and search for models in the Pareto front. We demonstrate the potential impact of our approach, COPA, in both model selection and benchmarking tasks across diverse ML areas such as fair ML, domain generalization, AutoML and foundation models, where classical ways to normalize and aggregate objectives fall short.

Primary Area: Evaluation (e.g., methodology, meta studies, replicability and validity, human-in-the-loop)

Submission Number: 13217