Avey Bidirectional Architecture

ICLR 2026 Conference Submission20729 Authors

Keywords: Bidirectional models, Transformer-based encoders

TL;DR: We propose a new bidirectional attention-free encoder

Abstract: Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention’s ability to deliver bidirectional contextualization with high parallelism, as popularized by BERT-style architectures. Recently, Avey was introduced as an autoregressive, attention-free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, including decoupled static and dynamic parameterizations, stability-oriented normalization, and neural compression. Results show that this reformulated architecture compares favorably to four widely used Transformer-based encoders, consistently outperforming them on standard token-classification and information-retrieval benchmarks while scaling more efficiently to long contexts.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 20729