What does GPT store in its MLP weights? A case study of long-range dependencies
Primary Area: visualization or interpretation of learned representations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Mechanistic Interpretability
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Large Language Models such as GPT are able to recall factual information about the world that they have learnt during training. This information must be stored in the model weights yet there is much we do not know about exactly what information is stored, where it is located and how it is retrieved. In this paper, we test and develop existing theories about information storage and retrieval through the example of bracketed sentences. We show that, in the case of recognizing brackets, where a model must learn during training to associate matching opening and closing brackets, very early multi-layer perceptron (MLP) layers in the source position are responsible for this association. This supports existing work on the importance of MLP layers as key-value memory stores (Meng et al., 2023) and a potential hierarchy of roles within transformers, whereby early layers are responsible for storing and retrieving lower level information compared to more abstract information which is stored in later layers (Geva et al., 2021).
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3868
Loading