Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning

Christian Raymond

Abstract:

In this paper, we develop upon the emerging topic of loss function learning, which aims to learn loss functions that significantly
improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for learning modelagnostic loss functions via a hybrid neuro-symbolic search approach. The framework first uses evolution-based methods to search
the space of primitive mathematical operations to find a set of symbolic loss functions. Second, the set of learned loss functions
are subsequently parameterized and optimized via an end-to-end gradient-based training procedure. The versatility of the proposed
framework is empirically validated on a diverse set of supervised learning tasks. Results show that the meta-learned loss functions
discovered by the newly proposed method outperform both the cross-entropy loss and state-of-the-art loss function learning methods
on a diverse range of neural network architectures and datasets