Differentiable model selection for ensemble learning

James Kotary, Vincenzo Di Vito Francesco, Ferdinando Fioretto

Abstract: Model selection is a strategy aimed at creating ac- curate and robust models. A key challenge in de- signing these algorithms is identifying the optimal model for classifying any particular input sample. This paper addresses this challenge and proposes a novel framework for differentiable model selection integrating machine learning and combinatorial op- timization. The framework is tailored for ensemble learning, a strategy that combines the outputs of in- dividually pre-trained models, and learns to select appropriate ensemble members for a particular in- put sample by transforming the ensemble learning task into a differentiable selection program trained end-to-end within the ensemble learning model. Tested on various tasks, the proposed framework demonstrates its versatility and effectiveness, out- performing conventional and advanced consensus rules across a variety of settings and learning tasks.