Structured Identity Mapping Learning As a Model for Compositional Generalization in Generative Models

Yongyi Yang, Core Francisco Park, Ekdeep Singh Lubana, Maya Okawa, Wei Hu, Hidenori Tanaka

TL;DR: We study the learning dynamics of a structured identity mapping learning task and show that it reveals the underlying mechanisms of some phenomenons observed in concept learning dynamics.

Abstract: Multi-modal generative models demonstrate complex concept learning dynamics, initially learning individual concepts and later recombining them in novel ways not present in the training data. Despite the practical importance of understanding the causal mechanisms underlying these learning dynamics, our theoretical understanding remains limited. In this work, we aim to bridge this gap by systematically analyzing the learning dynamics of a simplified model: a one-hidden-layer network learning the identity map, with a training set composed of Gaussian point clouds non-uniformly distributed in space. We argue that a simple yet describe model of multi-modal generative model is the task of learning identity mapping.

Style Files: I have used the style files.

Debunking Challenge: This submission is an entry to the debunking challenge.

Submission Number: 48