GeoCon: Compositional Generalization Through Geometric Constraints on Representation Structure

Chenming Shang, Shiji Zhou, Hengyuan Zhang, Xinchen Zhang, Lei Ke, Yuwang Wang, Yujiu Yang

Keywords: Represention Learning, Compositional Generalization.

Abstract:

Compositional generalization, referring to the capacity to generalize novel combinations of fundamental and essential concepts, is thought to be the mechanism underlying a human’s remarkable ability of rapid generalization to new knowledge and tasks. Recent research on brain neural activation space has found that the geometric structure of neural representations is highly related to human compositional generalization capability.
In this paper, we extend the above observations from neuroscience to deep neural networks to validate the potential relationship between the geometric structure of representations and compositional generalization capability. In particular, we first construct a new compositional generalization benchmark from the existent datasets, which aims to discriminate multiple concepts simultaneously through a powerful representation. Meanwhile, for the aforementioned geometric constraint, the parallelism score is formally defined for deep neural networks.
Subsequently, we decompose the deep neural network into two parts: the featurizer and the classifier, to investigate the relationship between compositional generalization capability and parallelism score separately. Our proposed method, Geometric Constraint (GeoCon), involves distance variance minimization on the classifier and parallelism score maximization on the featurizer.
Experiments on synthetic and real-world datasets demonstrate significant improvement of our approach, verifying the effectiveness of our neuroscience-inspired GeoCon approach towards human-like superior generalization ability.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4422