TOAST : Transformer Optimization using Adaptive and Simple Transformations

ICLR 2026 Conference Submission20193 Authors

Keywords: latent representations, representation learning, neural network similarities, classification, foundation models, transformers

TL;DR: We propose TOAST, a framework that approximates transformer blocks using simple transformations, reducing parameter count and GFLOPs while increasing throughput, with minimal to no impact on downstream task performance.

Abstract: Foundation models achieve state-of-the-art (SOTA) performance across different tasks, but their size and computational demands raise concerns about accessibility and sustainability. Existing efficiency methods often require additional retraining or fine-tuning, limiting their practicality. Recent findings suggest that deep neural networks exhibit internal representation similarities. While such similarities across different models have been exploited for enabling techniques such as model stitching and merging, intra-network redundancy remains underexplored as a source for efficiency gains. In this paper, we introduce Transformer Optimization using Adaptive and Simple Transformations (TOAST), a framework that exploits these redundancies to approximate entire transformer blocks with lightweight closed-form mappings, such as linear transformation or even the identity, without any additional training. Across SOTA pretrained vision models (e.g., ViT, DINOv2, DeiT) and datasets ranging from MNIST to ImageNet-1k, TOAST reduces parameters and computation while preserving, and in some cases improving, downstream performance. These results show that large portions of transformer depth can be replaced by trivial functions, opening a new perspective on efficient foundation models.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 20193