Knowledge Augmentation: In-context or In-parameter?
Keywords: Representation Learning, In-Parameter Knowledge Injection, Parametric Knowledge Representation, Large Language Models
TL;DR: This paper propose a novel in-parameter knowledge injection method and compare both in-context and in-parameter knowledge injection method for generative language models across a wide range of tasks.
Abstract: Large Language Models (LLMs) have achieved remarkable performance in various natural language processing tasks by leveraging relevant external knowledge provided by the users or retrieved from external sources.
Traditionally, this external information is incorporated by appending it directly to the model’s input context, a paradigm known as in-context knowledge injection.
However, this paradigm faces significant limitations due to the finite input context length of LLMs and often results in shallow integration between the external knowledge and the model’s internal representations.
To address the limitations of in-context knowledge injection, we propose a new knowledge injection paradigm called in-parameter knowledge injection, which temporarily embeds the external knowledge relevant to the user’s input directly into the model’s parameters rather than its input context.
This new paradigm overcomes the context length limitations of LLMs and enables deeper integration of external information within the model’s internal representations.
Through extensive experiments across tasks of varying complexity, we demonstrate that in-parameter knowledge injection achieves significant benefits for complex tasks requiring intricate reasoning.
In contrast, in-context injection remains effective for simpler tasks where answers can be directly extracted from the provided information.
We have open-sourced all the code, data, and models in the following anonymous GitHub link: https://anonymous.4open.science/r/In-parameter-Knowledge-Injection/
Supplementary Material: zip
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6880
Loading