Towards Improved Sentence Representations using Token Graphs

ICLR 2026 Conference Submission18979 Authors

Keywords: Graph-based token pooling; Sentence embeddings

TL;DR: A lightweight relational learning based pooling module for sentence representations from frozen LLMs. It attains SOTA on GLUE, IMDB, and MTEB, and remains highly robust to random noise.

Abstract: Obtaining a single-vector representation from a Large Language Model's (LLM) token-level outputs is a critical step for nearly all sentence-level tasks. However, standard pooling methods like mean or max aggregation treat tokens as an independent set, discarding the rich relational structure captured by the model's self-attention layers and making them susceptible to signal dilution. To address this, we introduce GLOT, a lightweight, structure-aware pooling module that reframes pooling as relational learning followed by aggregation. Operating on the outputs of a frozen LLM, GLOT first constructs a latent token-similarity graph, then refines token representations with a graph neural network, and finally aggregates them using a readout layer. Experimentally, our approach is remarkably robust and efficient: on a diagnostic stress test where 90% of tokens are random distractors, GLOT maintains over 97% accuracy while baseline methods collapse. Furthermore, it achieves state-of-the-art performance on benchmarks like GLUE and MTEB with 20x fewer trainable parameters and speeds up the training time by over 100x compared with parameter-efficient fine-tuning methods. Supported by a theoretical analysis of its expressive power, our work shows that learning over token graphs is a powerful paradigm for the efficient adaptation of frozen LLMs.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 18979