SynDoc: A Hybrid Discriminative-Generative Framework for Enhancing Synthetic Domain-Specific Visually-Rich Document Understanding

ACL ARR 2025 May Submission4520 Authors

Abstract: Domain-specific Visually Rich Document Understanding (VRDU) presents significant challenges due to the complexity and sensitivity of documents in fields such as medicine, finance, and material science. Existing Large (Multimodal) Language Models (LLMs/MLLMs) achieve promising results but face limitations such as hallucinations, inadequate domain adaptation, and reliance on extensive fine-tuning datasets. This paper introduces SynDoc, a novel framework that combines discriminative and generative models to address these challenges. SynDoc employs a robust synthetic data generation workflow, using structural information extraction and domain-specific query generation to produce high-quality annotations. Through adaptive instruction tuning, SynDoc improves the discriminative model's ability to extract domain-specific knowledge. At the same time, a recursive inferencing mechanism iteratively refines the output of both models for stable and accurate predictions. This framework demonstrates scalable, efficient, and precise document understanding and bridges the gap between domain-specific adaptation and general world knowledge.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: Visually Rich Document, Multimodal Large Language Model, Synthetic Data

Contribution Types: NLP engineering experiment

Languages Studied: English, Chinese

Submission Number: 4520