Enhancing LLMs via Lightweight Question-Attended Span Extraction

Anonymous

Abstract: To address the hallucination challenge in zero-shot LLMs without extensive task-specific prompt engineering, we introduce a lightweight Question-Attended Span Extraction (QASE) module during the fine-tuning of LLMs. Our experiments demonstrate that QASE empowers smaller models to outperform SOTA LLMs on reading comprehension tasks, notably achieving up to a 32.6% improvement over GPT-4's F1 score on SQuAD, all without increasing computational costs.

Paper Type: short

Research Area: Question Answering

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.