Enhancing text understanding of decoder-based model by leveraging parameter-efficient fine-tuning method

Wasif Feroze, Shaohuan Cheng, Elias Lemuye Jimale, Abdul Naveed Jakhro, Hong Qu

Abstract: Machine reading comprehension (MRC) is a fundamental natural language understanding task in natural language processing, which aims to comprehend the text of a given passage and answer questions based on it. Understanding implicit information, deducing the logical structure of information, and connecting context from different pieces of information make the MRC task difficult. Most current state-of-the-art approaches for MRC are using encoder-based models. However, no earlier research proposed a decoder-only model for MRC question-answering datasets, although language models based on this category achieved unprecedented performance in different generative tasks. In this paper, we propose a parameter-efficient fine-tuning framework that effectively increases MRC capabilities on decoder-only large language models. This framework designs the process for MRC and introduces the low-rank adaptation (LoRA) method to effectively fine-tune the large model with many parameters, even with lower hardware resource requirements than the previous methods. In addition, we also integrate a quantized model inference strategy for the fine-tuned model to improve practicability further. We conducted experiments on four types of MRC datasets. After extensive experiments, our results show that our model achieved a significant performance boost over baselines and outperformed other strong models for MRC.