Nayana: A Foundation for Document-Centric Vision-Language Models via Multi-Task, Multimodal, and Multilingual Data Synthesis

Adithya S Kolavi, Samarth P, Vyoman Jain

Keywords: Document Understanding, Vision-Language Models, Synthetic Data Generation, Document Retrieval, Optical Character Recognition, Visual Question Answering (VQA)

TL;DR: Nayana is a synthetic dataset of 3M annotated document images across 22 languages with preserved layouts to advance document-centric vision-language models for OCR, VQA and retrieval tasks.

Abstract:

We present Nayana, a comprehensive, synthetically generated dataset designed to advance document-centric vision language models across multiple tasks and languages. Nayana consists of three interconnected subsets, each targeting different aspects of document understanding: (1) a base dataset of 3 million document images with hierarchical annotations including detailed layout information, textual content, reading order, and relationships between document elements; (2) a multilingual variant spanning 22 languages, preserving the original document layout while translating text through contextual models; and (3) a specialized information retrieval subset for document ranking tasks with approximately 250,000 image-query pairs per language.
What distinguishes Nayana is its synthetic generation methodology. We collect a diverse corpus of PDFs from multiple sources, then apply state-of-the-art models to hierarchically extract structural and textual information, yielding a highly structured representation capturing layout elements, text lines, images, captions, and their interrelationships. For the multilingual extension, we employ contextual translation models to transform textual elements while preserving stylistic and visual attributes. Beyond the primary subsets, Nayana incorporates Visual Question Answering (VQA) pairs in both monolingual and multilingual settings. This multifaceted approach makes Nayana a truly multi-task dataset, enabling training of vision-language models for diverse applications including layout detection, equation recognition, image captioning, markdown conversion, multilingual OCR, document retrieval, and more.

Submission Number: 20