On Margins and Generalisation for Voting Classifiers

Felix Biggs, Valentina Zantedeschi, Benjamin Guedj

Keywords: PAC-Bayes, Generalisation bounds, Ensemble learning, Margins, Majority votes, Aggregation of experts

Abstract: We study the generalisation properties of majority voting on finite ensembles of classifiers, proving margin-based generalisation bounds via the PAC-Bayes theory. These provide state-of-the-art guarantees on a number of classification tasks. Our central results leverage the Dirichlet posteriors studied recently by Zantedeschi et al. (2021) for training voting classifiers; in contrast to that work our bounds apply to non-randomised votes via the use of margins. Our contributions add perspective to the debate on the ``margins theory'' proposed by Schapire et al. (1998) for the generalisation of ensemble classifiers.

TL;DR: A new margin bound for majority voting of weighted ensembles provides consistently tight empirical generalisation guarantees on real tasks.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2206.04607/code)