When are equilibrium networks scoring algorithms?

Russell Tsuchida, Cheng Soon Ong

Keywords: principal component analysis, deep equilibrium models, implicit neural network, exponential families

TL;DR: We derive an deep equilibrium network that finds MAP estimates of exponential family PCA, helping to place deep equilibrium networks in an unsupervised setting on a statistical foundation.

Abstract: Principal Component Analysis (PCA) and its exponential family extensions have three components: observed variables, latent variables and parameters of a linear transformation. The likelihood of the observation is an exponential family with canonical parameters that are a linear transformation of the latent variables. We show how joint maximum a-posteriori (MAP) estimates can be computed using a deep equilibrium model that computes roots of the score function. Our analysis provides a systematic way to relate neural network activation functions back to statistical assumptions about the observations. Our layers are implicitly differentiable, and can be fine-tuned in downstream tasks, as demonstrated on a synthetic task.

Student Paper: No