Learning Beyond the Surface: How Far Can Continual Pre-Training with LoRA Enhance LLMs' Domain-Specific Insight Learning?

ACL ARR 2025 May Submission257 Authors

Abstract: Large Language Models (LLMs) have demonstrated remarkable performance on various tasks, yet their ability to extract and internalize deeper insights from domain-specific datasets remains underexplored. In this study, we investigate how continual pre-training (CPT) can enhance LLMs' capacity for insight learning across three distinct forms: declarative, statistical, and probabilistic insights. Focusing on two critical domains: medicine and finance, we employ LoRA to train LLMs on two existing datasets. To evaluate each insight type, we create benchmarks to measure how well continual pre-training (CPT) helps models go beyond surface-level knowledge. We also assess the impact of document modification on capturing insights. The results show that, while CPT on original documents has a marginal effect, modifying documents to retain only essential information significantly enhances the insight-learning capabilities of LLMs. We will release our dataset and code.

Paper Type: Short

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: parameter-efficient-training, benchmarking, pre-training, insight mining, document-level extraction

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources

Languages Studied: English

Submission Number: 257