Yuan Ye ; Liu Xiao ; Dikubab Wondimu ; Liu Hui ; Ji Zhilong ; Wu Zhongqin ; Bai Xiang

Abstract

Handwritten mathematical expression recognition (HMER) is a challenging taskthat has many potential applications. Recent methods for HMER have achievedoutstanding performance with an encoder-decoder architecture. However, thesemethods adhere to the paradigm that the prediction is made "from one characterto another", which inevitably yields prediction errors due to the complicatedstructures of mathematical expressions or crabbed handwritings. In this paper,we propose a simple and efficient method for HMER, which is the first toincorporate syntax information into an encoder-decoder network. Specifically,we present a set of grammar rules for converting the LaTeX markup sequence ofeach expression into a parsing tree; then, we model the markup sequenceprediction as a tree traverse process with a deep neural network. In this way,the proposed method can effectively describe the syntax context of expressions,alleviating the structure prediction errors of HMER. Experiments on threebenchmark datasets demonstrate that our method achieves better recognitionperformance than prior arts. To further validate the effectiveness of ourmethod, we create a large-scale dataset consisting of 100k handwrittenmathematical expression images acquired from ten thousand writers. The sourcecode, new dataset, and pre-trained models of this work will be publiclyavailable.