ModernVBERT: Towards Smaller Visual Document Retrievers

ICLR 2026 Conference Submission18771 Authors

Keywords: Visual Document Retrieval, Multimodal Encoder, Late Interaction, Document Embeddings

TL;DR: We revisit decisions in the training process of visual document retrievers and demonstrate our findings by releasing a small model that outperforms 10x bigger models on benchmarks.

Abstract:

Retrieving specific information from a large corpus of documents is a widely prevalent industrial use case of modern AI, notably due to the popularity of Retrieval-Augmented Generation (RAG) systems. Although neural document retrieval models have historically operated exclusively in the text space, Visual Document Retrieval (VDR) models - large vision–language decoders repurposed as embedding models which directly work with page screenshots as inputs - are increasingly popular due to the performance and indexing latency gains they offer. In this work, we show that, while cost-efficient, this approach of repurposing generative models bottlenecks retrieval performance.

Through controlled experiments, we revisit the entire training pipeline, and establish a principled recipe for improving visual document retrieval models. We notably measure the impact of attention masking, image resolution, modality alignment data regimes, and late interaction centered contrastive objectives which emerge as central performance factors.
Building on these insights, we release ModernVBert, a compact 250M parameter vision–language encoder that outperforms recent models up to 10 times larger when fine-tuned on document retrieval tasks, enabling efficient inference on cheap CPU hardware and greatly reducing latency and costs while maintaining strong performance. Models, code and data are available in the public version of this work under an open license.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 18771