Layer-Aware Embedding Fusion for Text Classification with LLMs

ACL ARR 2025 May Submission2256 Authors

Abstract: Embedding fusion has emerged as an effective approach for enhancing performance across various NLP tasks. However, systematic guidelines for selecting optimal layers and developing effective fusion strategies for the integration of LLMs remain underexplored. In this study, we propose a layer-aware embedding selection method and investigate how to quantitatively evaluate different layers to identify the most important ones for downstream NLP tasks, showing that the critical layers vary depending on the dataset. We also explore how combining embeddings from multiple LLMs, without requiring model fine-tuning, can improve performance. Experiments on four English text classification datasets (SST-2, MR, R8, and R52) demonstrate that different layers in LLMs exhibit varying degrees of representational strength for classification, and that combining embeddings from different models can enhance performance if the models exhibit complementary characteristics. Additionally, we discuss resources overhead (memory and inference time) to provide a balanced perspective on the real-world feasibility of embedding fusion.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: Representation learning,Transfer learning / domain adaptation

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 2256