A Nested Ensemble Method to Bilevel Machine Learning

Lisha Chen, Momin Abbas, Tianyi Chen

Abstract: Modern machine learning problems, such as hyperparameter optimization, meta learning, and adversarial training, adopt a bilevel learning formulation. Such problems involve a nested relation between inner- and outer-level problems, which often have suboptimal solutions with poor generalization ability. To address this issue, this paper proposes an ensemble method tailored to bilevel learning. Our method finds a nested ensemble of inner and outer parameters that improve generalization. We instantiate our general results with meta learning. We show theoretically and empirically that the diversity and the smoother loss landscape of the proposed ensemble methods lead to improved generalization over the state-of-the-art method.