A Comprehensive Survey of Loss Functions in Machine Learning

Qi Wang, Yue Ma, Kun Zhao, Yingjie Tian

Abstract:

As one of the important research topics in machine learning, loss function plays an
important role in the construction of machine learning algorithms and the improvement of their performance, which has been concerned and explored by many
researchers. But it still has a big gap to summarize, analyze and compare the classical loss functions. Therefore, this paper summarizes and analyzes 31 classical loss
functions in machine learning. Specifcally, we describe the loss functions from the
aspects of traditional machine learning and deep learning respectively. The former is
divided into classifcation problem, regression problem and unsupervised learning
according to the task type. The latter is subdivided according to the application scenario, and here we mainly select object detection and face recognition to introduces
their loss functions. In each task or application, in addition to analyzing each loss
function from formula, meaning, image and algorithm, the loss functions under the
same task or application are also summarized and compared to deepen the understanding and provide help for the selection and improvement of loss function.