Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledge

Sahil Kale

Keywords: Memorization, Self-Knowledge, Large language models

TL;DR: LLMs draw unnaturally high confidence from memorized patterns to infer an inflated self-knowledge about their reasoning ability, manifesting as an over 45% inconsistency in feasibility assessments when faced with logically coherent task perturbations

Abstract: When artificial intelligence mistakes memorization for intelligence, it creates a dangerous perception of reasoning. Existing studies treat memorization and self-knowledge deficits in LLMs as separate issues. In our study, we pinpoint an intertwined causal link between the two that undermines the trustworthiness of LLM responses. To investigate this, we utilize a novel framework to ascertain if LLMs genuinely learn reasoning patterns from training data or merely memorize them to assume competence across problems of similar complexity focused on STEM domains. Our analysis shows a noteworthy problem in generalization: LLMs draw confidence from memorized solutions to infer a higher self-knowledge about their reasoning ability, which manifests as an over 45% inconsistency in feasibility assessments when faced with self-validated, logically coherent task perturbations. This effect is most pronounced in science and medicine domains. Our code and results are available publicly at https://anonymous.4open.science/r/LLM-Memorization_SK_Eval--543D/.

Submission Number: 1