Knowledge Manipulation in Language Models (Part B)

Zeyuan Allen-Zhu, Yuanzhi Li

Keywords: Interpretability, Transformers, Language Models, Linear Probing, Inner Working, Factual Knowledge

TL;DR: Why do LLMs need Chain of Thoughts even for basic questions (e.g. was Biden born on an even day)? We show that LLMs cannot efficiently manipulate knowledge even if such knowledge is 100% extractable; plus, inverse knowledge search is just impossible.

Abstract:

Language models can store vast amounts of factual knowledge, but their ability to use this knowledge for logical reasoning remains questionable.
This paper explores a language model's ability to manipulate its stored knowledge during inference. We focus on four manipulation types: *retrieval* (e.g., "What is person A's attribute X"), *classification* (e.g., "Is A's attribute X even or odd?"), *comparison* (e.g., "Is A greater than B in attribute X?") and *inverse search* (e.g., "Which person's attribute X equals T?")

We observe that pre-trained language models like GPT2/3/4 excel in knowledge retrieval but struggle with simple classification or comparison tasks unless Chain of Thoughts (CoTs) are employed during both training and inference. They also perform poorly in inverse knowledge search, irrespective of the prompts. Our primary contribution is a synthetic dataset for a *controlled experiment* that confirms these inherent weaknesses: a language model cannot *efficiently* manipulate knowledge from pre-training data, even when such knowledge is perfectly stored and fully extractable in the models, and despite adequate instruct fine-tuning.

Primary Area: visualization or interpretation of learned representations

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 736