Tell Me What To Learn: Generalizing Neural Memory to be Controllable in Natural Language

Max S. Bennett, Thomas P Zollo, Richard Zemel

Keywords: neural memory, continual learning, lifelong learning

TL;DR: We introduce a generalized neural memory system that uses natural-language learning instructions to selectively update what a model remembers or ignores over time, enabling efficient and controlled continual adaptation.

Abstract: Modern machine learning models are deployed in diverse, non-stationary environments where they must continually adapt to new tasks and evolving knowledge. Continual fine-tuning and in-context learning are costly and brittle, whereas neural memory methods promise lightweight updates with minimal forgetting. However, existing neural memory models typically assume a single fixed objective and homogeneous information streams, leaving users with no control over what the model remembers or ignores over time. To address this challenge, we propose a generalized neural memory system that performs flexible updates based on learning instructions specified in natural language. Our approach enables adaptive agents to learn selectively from heterogeneous information sources, supporting use-cases—such as healthcare and customer service—where fixed-objective memory updates are insufficient.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 1