A Gating Layer-Based Restorable Embedding Framework for Efficient Knowledge Representations

Anonymous

Abstract: Large language models have achieved linguistic fluency and exhibited remarkable performances in various natural language tasks without gradient updates because more number of model parameters could retain more knowledge. However, large language models are not applicable to the domain-specific tasks requiring knowledge not included in the training corpus, due to the fact that knowledge in the model parameters is not controllable during generation and updating the model parameters is costly. This research introduces efficient embedding mechanisms to separate knowledge from language models. The method divides the previous end-to-end construction of the language models into three sub-parts: sentence-level knowledge encoding, sentence-embedding-based task processing, and restoring the processed knowledge embedding to token-level embedding. The experimental results verify that most knowledge consisting of 1 or 2 sentences can be restored and the performance in the passage retrieval task is significantly improved.e

Paper Type: long

Research Area: Efficient/Low-Resource Methods for NLP

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.