Improving LLM Predictions via Inter-Layer Structural Encoders

Tom Ulanovski, Eyal Blyachman, Maya Bechler-Speicher

Track: tiny paper (up to 4 pages)

Keywords: Large Language Models, Inter-Layer, Geometry, Representations refining, Graph Neural Network, Embeddings

Abstract:

Standard LLM inference relies on final-layer representations, despite evidence that intermediate layers often capture task-specific information more effectively. However, identifying the optimal layer remains task-dependent and computationally
expensive. In this work we introduce Inter-Layer Structural Encoders (ILSE), an approach to learn one representation from the LLM’s internal layer representations all together. Central to ILSE is Cayley-Encoder, a geometric encoder which builds upon recent studies leveraging Cayley Graphs for neural information propagation. We evaluate ILSE across 13 classification and semantic similarity tasks with 2 pretrained LLMs. ILSE consistently outperforms baselines and existing approaches, achieving up to 40% improvement in accuracy and 22% in similarity metrics.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 41