Simple mechanisms for representing, indexing and manipulating concepts

Yuanzhi Li, Raghu Meka, Rina Panigrahy, Kulin Shah

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: learning theory, theory of representations, manifolds

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We introduce a simple sketch for defining concepts mathematically and building recursive structures out of these concepts.

Abstract:

Deep networks typically learn concepts via classifiers, which involves setting up a model and training
it via grading descent to fit the concept-labeled data. We will argue instead that learning a concept could
be done by looking at its moment statistics matrix to generate a concrete representation or signature of
that concept. These signatures can be used to discover structure across the set of concepts and could
recursively produce higher-level concepts by learning this structure from those signatures. Concepts can
be ’intersected’ to find a common theme in a number of related concepts. This process could be used to
keep a dictionary of concepts so that inputs could correctly identify and be routed to the set of concepts
involved in the (latent) generation of the input.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7534