Can Decoding by Contrasting Layers Really Improve Factuality in Large Language Models?

Ming Liu, Wensheng Zhang

Keywords: Factuality, Contrastive Decoding, Parametric Memory

Abstract:

Large language models (LLMs) have made notable advancements across diverse applications, but their susceptibility to hallucinations remains a critical challenge. That is, they could produce outputs divergent from real-world evidence or user-provided inputs. Recent studies have explored a contrastive decoding strategy known as DoLa, which mitigates output inaccuracy by contrasting the outputs from the final layer against those from the previous layers. Nevertheless, such strategy has its limitation, as LLMs, which already have internalized extensive parametric knowledge through comprehensive pre-training and fine-tuning phases, may generate errors due to incorrect or obsolete information within their parameters. As an alternative, trusted external knowledge could be included in the prompt context for querying, but the constrained context window of LLMs poses a significant barrier restricting the amount of information that can be provided.
To address the above issues, we propose to integrate the contrasive decoding strategy with a long-context encoder that effectively condenses extensive initial contexts into a more concise format. Extensive experiments have demonstrated that,
our proposed methodology enhances the factual accuracy of the produced content,
when applied to various datasets. For instance, it has improved the performance of LLaMA2-7B models on the Quality dataset by 61.61\%, compared to the DoLa decoding method, showcasing its effectiveness in enhancing the reliability of LLMs in generating truthful information.

Primary Area: other topics in machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12305