Effective Regularization Through Loss-Function Metalearning

Santiago Gonzalez, Risto Miikkulainen

Keywords: regularization, loss, loss function, metalearning, meta-learning, optimization, theory, robustness, adversarial attacks

Abstract: Loss-function metalearning can be used to discover novel, customized loss functions for deep neural networks, resulting in improved performance, faster training, and improved data utilization. A likely explanation is that such functions discourage overfitting, leading to effective regularization. This paper theoretically demonstrates that this is indeed the case: decomposition of learning rules makes it possible to characterize the training dynamics and show that loss functions evolved through TaylorGLO regularize both in the beginning and end of learning, and maintain an invariant in between. The invariant can be utilized to make the metalearning process more efficient in practice, and the regularization can train networks that are robust against adversarial attacks. Loss-function optimization can thus be seen as a well-founded new aspect of metalearning in neural networks.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: This paper provides a theoretical foundation to explain how and why metalearned loss functions are able to regularize.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2010.00788/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=O6KU8mD-qw