RoBlock: Wide and Deep Scaling of Recommenders via Embedding Collapse Mitigation

Guoming Li, Jian Yang, Junwei Pan, Shangyu Zhang, Yi.Yang, CHAO ZHOU, Shudong Huang, Haijie Gu, Yifei Zhang, Menglin Yang

Keywords: Embedding Collapse, Recommendation Models, Rank Analysis, Dimensional Robustness, Spectrum Analysis, CTR Prediction

TL;DR: We propose RoBlock, a building block that mitigates both depth- and width-wise embedding collapse in recommendation models, producing dimensionally robust embeddings and consistently improving performance as models scale.

Abstract:

Scaling recommendation models has emerged as a promising direction for advancing recommender systems, yet they face a fundamental challenge: *embedding rank collapse*.
This phenomenon, rooted in the intrinsic properties of feature interaction modules, causes embedding matrices to lose representational capacity as models scale, severely limiting their effectiveness.
Existing solutions primarily focus on width-wise scaling via *multi-embedding*, which parallelizes multiple embedding tables and has shown success in alleviating collapse.
However, these methods configure only the initial embedding layer and fail to address *depth-wise* embedding collapse, which intensifies with increasing model depth and restricts the benefits of deeper architectures.
We propose **RoBlock**, a stackable building block that delivers dimensionally-robust embeddings by mitigating collapse across width and depth.
RoBlock integrates three key components: (1) **spectrum rebalancing** through rank-1 update normalization to restore the spectrum distribution of embedding matrices, (2) an **embedding decoupler** guided by the Hilbert–Schmidt Independence Criterion (HSIC) to extract independent embedding components while preserving spectrum (dimensional) robustness, and (3) **embedding regeneration** via a field-wise multi-head router to regenerate non-collapsed embedding sets, achieving the benefits of multi-embedding within each block.
Theoretical analysis establishes that RoBlock effectively mitigates embedding collapse, providing a principled foundation for scalable recommendation models.
Extensive experiments across multiple datasets further demonstrate that RoBlock consistently alleviates embedding collapse across layers and delivers significant performance gains over the baselines, with improvements growing as model width and depth increase.
The code is accessible at the anonymous link: https://anonymous.4open.science/r/RoBlock-2F8A

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 23146