UniLoss: Unified Surrogate Loss by Adaptive Interpolation

Lanlan Liu, Mingzhe Wang, Jia Deng

Abstract: We introduce UniLoss, a unified framework to generate surrogate losses for training deep networks with gradient descent, reducing the amount of manual design of task-specific surrogate losses. Our key observation is that in many cases, evaluating a model with a performance metric on a batch of examples can be re-factored into four steps: from input to real-valued scores, from scores to comparisons of pairs of scores, from comparisons to binary variables, and from binary variables to the final performance metric. Using this re-factorization we generate a unified differentiable approximation of the evaluation computation, through adaptive interpolation at selective configurations of the binary variables. Using UniLoss, we can optimize for different tasks and metrics using one unified framework, achieving comparable performance compared with task-specific losses. We validate the effectiveness of UniLoss on three tasks and four datasets.