Teaching VLMs to Admit Uncertainty in OCR from Lossy Visual Inputs

Shuhao Guan, Moule Lin, Cheng Xu, Jinman Zhao, Derek Greene

Keywords: Optical Character Recognition, Visually Degraded Document, Uncertainty, LLM

TL;DR: We propose uncertainty-aware OCR, our model transcribes while explicitly bracketing spans it deems unreliable with uncertainty tags.

Abstract: Vision-language models (VLMs) are increasingly replacing traditional OCR pipelines. However, they often hallucinate on lossy visual inputs, such as visually degraded document images, producing fluent yet incorrect text without signaling uncertainty. This occurs because current post-training emphasizes accuracy, which encourages models to guess even when uncertain. The problem persists in state-of-the-art systems and severely impacts OCR reliability. To improve the trustworthiness of OCR on degraded documents, we propose uncertainty-aware OCR. Rather than suppressing guesses, our model transcribes while explicitly bracketing spans it deems unreliable with uncertainty tags. To train our model, we use Group Relative Policy Optimization (GRPO). We define usage rules for uncertainty tags and an evaluation protocol, introducing a pseudo-labeled cold start and a multi-objective reward that balances transcription accuracy and uncertainty coverage while preventing reward hacking. We explore different combinations of cold-start and reward granularity. We also assess the effect of reward parameters in preventing reward hacking and improving the corresponding metrics. Furthermore, we introduce Blur-OCR, a challenging benchmark for uncertainty-aware OCR on degraded document images under lossy visual conditions. In extensive experiments, our model maintains transcription accuracy while achieving an uncertainty tag F1 score of 0.685.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 1052