Layer-Centric Factors of Variation Disentanglement for Task- and Model-Agnostic Generalization

Hee-Jun Jung, Jongmin Park, Minwoo Kang, Hoyong Kim, Kangil Kim

Abstract:

Disentanglement learning aims to separate the underlying factors of variation (FoV) to improve generalization.
However, most FoV-based latent-vector-centric methods impose objective-driven constraints at a bottleneck, and it is difficult to translate disentanglement into consistent gains on downstream tasks without inductive bias.
Motivated by architectural approaches complementary to vector-centric objectives for downstream tasks, we propose the *Orthogonal Subspaces Projection* (OSP) layer, a plug-and-play module that integrates into intermediate layers and promotes FoV separation by projecting latent features into mutually orthogonal subspaces.
Across diverse domains and tasks, models equipped with the OSP layer improve disentanglement quality and generalization in downstream tasks, including computer vision (classification, detection, and segmentation), natural language processing (word analogy, and text classification), and fine-tuning settings on large backbones.

Lay Summary:

Modern AI systems often perform well on familiar data but struggle when important details appear in new combinations, such as a known object in a new scene or with a different visual attribute. A common way to address this is to help models separate the different factors that explain the data, but many existing methods try to do this only at the model’s final compressed representation, which does not always improve real-world performance.

We propose a simple plug-and-play layer called the *Orthogonal Subspaces Projection* (OSP) layer. Instead of forcing separation only at the end of a model, the OSP layer helps organize information throughout the model by placing different parts of the learned representation into separate, non-overlapping spaces. This is similar to sorting mixed information into different drawers so that each drawer can focus on a different kind of signal.

Because the OSP layer can be inserted into common neural network layers, it can be used with many existing models. Across image, language, and fine-tuning tasks, models with OSP layer show improved representation quality and better generalization with little extra cost.

Link To Code: https://github.com/GIST-IRR/OSP_layer

Primary Area: Deep Learning->Other Representation Learning

Keywords: Disentanglement

Submission Number: 30191