LOCR: Location-Guided Transformer for Optical Character Recognition

Anonymous

Abstract: Academic documents are packed with texts, equations, tables, and figures, requiring comprehensive understanding for accurate Optical Character Recognition (OCR). While end-to-end OCR methods offer improved accuracy over layout-based approaches, they often grapple with significant repetition issues, especially with complex layouts in Out-Of-Domain (OOD) documents. To tackle this, we propose LOCR, a model integrating location guiding into the transformer architecture during autoregression, training on a dataset comprising over 77M text-location pairs from 125K academic document pages, including bounding boxes for words, tables and mathematical symbols. LOCR adeptly handles various formatting elements and generates content in Markdown language. It outperforms existing models in our testset with an edit distance of 0.125, BLEU of 0.827 and F1 of 0.897.LOCR also reduces repetition frequency from 51% to 2% in the arXiv dataset and from 56% to 7% in OOD documents. Additionally, LOCR features an interactive OCR mode, facilitating the generation of complex documents through a few location prompts from human.

Paper Type: long

Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond

Contribution Types: Model analysis & interpretability, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English