Improving Autoencoder Performance on Sparse Binary Data through Sparsity-Aware Loss Functions

ICLR 2026 Conference Submission21591 Authors

Keywords: Autoencoders, sparse binary data

TL;DR: Incorporating sparsity-awareness into reconstruction losses enhances autoencoder fidelity, representation quality, and downstream utility on sparse binary data.

Abstract: Conventional reconstruction losses for autoencoders such as mean squared error (MSE) and binary cross-entropy (BCE) are poorly suited for sparse binary data. These measures can achieve deceptively low loss by trivially predicting the dominant zeros, while failing to capture the rare but informative non-zero entries. Prior work has primarily focused on architectural modifications or training heuristics to address this issue, leaving the design of loss functions largely overlooked. In this work, we shift focus to the reconstruction loss itself, exploring sparsity-aware reconstruction losses by extending focal loss, dice loss, and related formulations to the autoencoder setting. We evaluate their effect on both reconstruction fidelity and embedding quality across multiple sparse datasets, showing that these alternatives outperform MSE and BCE on metrics sensitive to rare events. Our results demonstrate that the choice of loss function is a critical but underappreciated factor in learning effective representations from sparse binary data.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 21591