Language Models Struggle to Explain Themselves
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: LLM, introspection, explainability, interpretability, self-explanation, honesty, faithfulness, truthfulness, classification, benchmark, evaluation, alignment, safety, dataset
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: This paper evaluates whether large language models can provide high-level explanations of their own internal processes by testing their ability to articulate simple classification rules used to solve text-based classification tasks.
Abstract: Large language models (LLMs) perform well at a myriad of tasks, but explaining the processes behind this performance is a challenge. This paper investigates whether LLMs can give faithful high-level explanations of their own internal processes. To explore this, we introduce a dataset, ArticulateRules, of few-shot text-based classification tasks generated by simple rules. Each rule is associated with a simple natural-language explanation. We test whether models that have learned to classify inputs competently (both in- and out-of-distribution) are able to articulate freeform natural language explanations that match their classification behaviour, using the simple rules as ground-truth explanations. Our dataset can be used for both in-context and finetuning evaluations. We evaluate a range of LLMs, demonstrating that articulation accuracy increases with model size, with a particularly sharp jump from GPT-3 to GPT-4. We then investigate whether we can improve GPT-3's articulation accuracy through finetuning. GPT-3 completely fails to articulate $7/10$ rules in our test, even after additional finetuning on correct explanations. We release our dataset, ArticulateRules, which can be used to test self-explanation for LLMs trained either in-context or by finetuning.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3317
Loading