Learning Single-Component Discriminative Representations via Maximal Coding Rate Reduction

Menghui Zhou, Gaoshan Bi, Vitaveska Lanfranchi, Po Yang

Keywords: linear discriminative representation, structured representation, coding rate, representation learning, rate reduction, low-dimensional structure

Abstract: The recently proposed maximal coding rate reduction principle (MCR$^2$) offers a promising theoretical framework for interpreting modern deep networks through the lens of data compression and discriminative representation. It maps high-dimensional multi-class data into mutually orthogonal linear subspaces, with each subspace capturing as many structural details of its class as possible. In this work, we show that such structural maximization not only increases model sensitivity to feature noise but also hinders generalization. In contrast, we argue that retaining only the single most discriminative structural component per class improves both generalization and robustness to feature noise, while preserving the desirable properties of MCR$^2$, such as robustness to label noise and resistance to catastrophic forgetting. We formalize this approach as a new framework termed SiMCoding and validate it extensively across supervised learning, white-box architectures, and incremental learning on diverse datasets. The superior performance of SiMCoding highlights its potential as a strong alternative for medium-scale classification tasks, particularly under label and feature noise.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 2838