End-to-end learning of energy-based representations for irregularly-sampled signals and images

Ronan Fablet, Lucas Drumetz, François Rousseau

TL;DR: We address the end-to-end learning of energy-based representations for signal and image observation dataset with irregular sampling patterns.

Abstract: For numerous domains, including for instance earth observation, medical imaging, astrophysics,..., available image and signal datasets often irregular space-time sampling patterns and large missing data rates. These sampling properties is a critical issue to apply state-of-the-art learning-based (e.g., auto-encoders, CNNs,...) to fully benefit from the available large-scale observations and reach breakthroughs in the reconstruction and identification of processes of interest. In this paper, we address the end-to-end learning of representations of signals, images and image sequences from irregularly-sampled data, {\em i.e.} when the training data involved missing data. From an analogy to Bayesian formulation, we consider energy-based representations. Two energy forms are investigated: one derived from auto-encoders and one relating to Gibbs energies. The learning stage of these energy-based representations (or priors) involve a joint interpolation issue, which resorts to solving an energy minimization problem under observation constraints. Using a neural-network-based implementation of the considered energy forms, we can state an end-to-end learning scheme from irregularly-sampled data. We demonstrate the relevance of the proposed representations for different case-studies: namely, multivariate time series, 2{\sc } images and image sequences.

Keywords: end-to-end-learning, irregularly-sampled data, energy representations, optimal interpolation

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:1910.00556/code)