Semantic-Aware Non-local Network for Handwritten Mathematical Expression Recognition

Xiang-Hao Liu, Da-Han Wang, Xia Du, Shunzhi Zhu

Abstract: Handwritten mathematical expression recognition (HMER) is a challenging task due to its complex two-dimensional structure of mathematical expressions and the high similarity between handwritten texts. Most existing encoder-decoder approaches for HMER mainly depend on local visual features but are seldom studied in explicit global semantic information. Besides, existing works for HMER primarily focus on local information. However, this obtained information is difficult to transmit between distant locations. In this paper, we propose a semantic-aware non-local network to tackle the above problems for HMER. Specifically, we propose to adopt the non-local network to capture long-term dependencies while integrating local and non-local features. Moreover, we customized the FastText language model to our backbone to learn the semantic-aware information. The experimental results illustrate that our design consistently outperforms the state-of-the-art methods on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014 and 2016 datasets.