Neural Predictive Text for Grammatical Error Prevention

Anonymous

Abstract: In this paper we study the potential of two neural language models, an LSTM and an autoregressive language model GPT-2, to predict possible correction tokens in erroneous sentences and to predict the next token in randomly sliced correct sentences, in the aim of establishing a new Grammatical Error Correction (GEC) subarea, for which we coin the term Grammatical Error Prevention (GEP). Systems that could assist in GEP, such as language models, are expected to predict elements and therefore prevent grammatical errors in advance. Our findings show that GPT-2 can predict 29% of the correct tokens with one prediction. Accuracy rises up to 44% when the top 3 predictions are considered. To test the pedagogical capacity of such a model, we also experimented with real English as a second language (ESL) learners. By equipping GPT-2 to generate text that functions as potential continuation of the learners' sentences, we created a small corpus of the learners' writings and analyzed their errors along with their frequencies.