Who's a Better Scholar: Encoder or Decoder?

ACL ARR 2025 February Submission8181 Authors

Abstract: Language modeling has seen seen a tremendous development over past few years, with a considerable rise in their deployment for solving domain-specific Natural Language Processing (NLP) tasks. In recent times, the fundamental building blocks of language models are essentially composed of either an encoder-based architecture or a decoder-based architecture or a combination of both. In the scholarly domain, the majority of use cases have explored only the utilization of encoder-only models for a variety of tasks using the pre-trained model fine-tuning approach. But the same has not yet been replicated for decoder based models in spite of the recent popularity of LLMs. To address this issue, we fine-tune both encoder-based language models and decoder-based language models on an array of traditional scholarly NLP tasks. This allows us to compare the effect of learned representations in contrast to generation-based techniques on standard scholarly benchmark datasets. We conduct extensive experiments on 10 highly popular human-annotated datasets over 6 different tasks and also study the effect of domain-specific pre-training on these tasks. We achieve SOTA over two tasks using decoder-based language models.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: Fine-tuning Language Models, Named Entity Recognition, Relation Extraction, Paraphrase Recognition, Natural Language Inference, Fact Checking

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 8181