Coupling RNNs with LLMs: Does Their Integration Improve Language Modeling Performance?

ACL ARR 2025 February Submission6 Authors

Abstract: Pretrained large language models (LLMs) have demonstrated remarkable success across various language modeling tasks. However, they continue to face inherent limitations in achieving state-of-the-art performance on many domain-specific applications. Previous research has explored diverse methodologies to enhance the performance of LLMs on downstream tasks. In this paper, we propose integrating recurrent neural networks (RNNs) with LLMs and investigate whether this integration improves language modeling performance. Particularly, LLMs are employed to generate rich and meaningful word embeddings, while RNNs excel at capturing the contextual semantics of long-range dependencies. The resulting LLM-RNN model leverages the complementary strengths of sequential and Transformer-based architectures to achieve enhanced performance. We conducted extensive experiments with rigorous hyperparameter tuning on multiple benchmark and real-world datasets. The experimental results highlight the superiority of the integrated LLM-RNN model in commonsense reasoning, code understanding, and biomedical reasoning tasks. Our codes are available at https://github.com/mostafiz26/CouplingRNNsLLMs.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: Coupling RNNs with LLMs, Language Modeling, LLM, RNN, Coupling, Biomedical Reasoning, Code Understanding, Commonsense Reasoning

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Reproduction study, Position papers

Languages Studied: Not applicable

Submission Number: 6