Go to ACL ARR 2025 February homepagePaperFormer: A Citation-Graph Enhanced Language Model for Scientific Applications
Abstract: The rapid expansion of scientific literature presents significant challenges in navigating, understanding, and utilizing scholarly knowledge effectively. To address this, we introduce PaperFormer, a citation-network-aware Language Model designed to enhance scientific tasks by incorporating citation graph information. PaperFormer augments a base model with additional specialized weights to effectively process and analyze research papers within their citation contexts. To support this research, we also release a novel dataset comprising approximately 10K papers, 42K reviews and rebuttals and 200K citation relationships. Our model undergoes pre-training on the Semantic Scholar Network (SSN) dataset and is evaluated across three tasks: causal language modeling, paper summarization, and automated review generation. Experimental results demonstrate that PaperFormer outperforms the state-of-the-art model in the paper summarization task and surpasses the base model in review generation. To foster further research, we open-source our models and the review-citations dataset, enabling broader adoption and extension of our work.
Paper Type: Long
Research Area: NLP Applications
Research Area Keywords: nlp applications, language modeling, generation
Contribution Types: NLP engineering experiment, Data resources
Languages Studied: English
Submission Number: 8187
Loading