Offline Handwritten Mathematical Expression Recognition via Graph Reasoning Network

Jia-Man Tang, Jin-Wen Wu, Fei Yin, Lin-Lin Huang

Abstract: Handwritten mathematical expression recognition (HMER) remains a challenge due to the complex 2-D structure layout and variable writing styles. Recently, large progress has been made by using deep encoder-decoder networks, which treat HMER as an Image-to-Sequence task and parse the math expression into a sequence (i.e. LaTeX). However, (1) mathematical expression is a 2-D structure pattern and sequence representation can not explicitly explore the structural relationship between symbols. (2) Image-to-Sequence as recurrent models can not infer in parallel during test stage. In this paper, we formulate mathematical expression recognition as an Image-to-Graph task and propose a Graph Reasoning Network (GRN) for offline HMER task. Compared with sequence representation, graph representation is more interpretable and more consistent with human visual cognition. Our method builds graph on math symbols detected from image, aggregates node and edge features via a Graph Neural Network (GNN) and parses the graph to give Symbol Layout Tree (SLT) format recognition result via node and edge classification. Experiments on public datasets show that our model achieve competitive results against other methods and can interpret the located symbols and inter-relationship explicitly.

External IDs:dblp:conf/acpr/TangWYH21