Structure-Aware Adapter for Large Language Model

Anonymous

Abstract: Pre-trained Large Language Models (LLMs) have been shown effective in various natural language processing tasks, especially when fine-tuned on specific downstream scenarios. However, the full fine-tuning of LLMs is usually computationally expensive and time-consuming due to the ever-increasing parameter size. In addition, while the LLMs are pre-trained to memorize the facts and knowledge from unstructured textual corpora, they cannot be well generalized to some domain-specific scenarios where additional structured knowledge is required, such as enterprise databases or social graphs. In this paper, we design a novel structure-aware adapter for LLMs to utilize structured relational information from knowledge graphs with a structure-aware relational attention mechanism. The proposed adapter framework only introduces a small scale of new parameters and therefore significantly reduces the cost of fine-tuning, without perturbing the initial pre-trained parameters of LLMs. We also propose a knowledge-graph-induced path-of-thought prompt to enhance the utilization of the LLM adapter to retrieve information from the knowledge graph. We evaluate the proposed model on two question-answering benchmarks. The evaluation results show that the proposed method outperforms the state-of-the-art LLM adapters by 4.1%-15.9% and 1.4%-17.6% in question-answering accuracy of CSQA and OBQA datasets. Ablation studies are also discussed to prove the effectiveness of the proposed modules.

Paper Type: long

Research Area: Efficient/Low-Resource Methods for NLP

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency, Theory

Languages Studied: English