Microtext Normalization for Chatbots

Ranjan Satapathy, Erik Cambria, Nadia Magnenat-Thalmann

Abstract: With the current upsurge in the usage of social media platforms, the trend of using short text, or microtext, in place of standard English has witnessed a significant rise. This work incorporates microtext normalization into a robot’s chatbot. The work leverages the fact that humans tend to write in different unconstrained ways. This work also involves a binary classifier to detect microtext, which helps in reducing the execution time of the microtext normalization module. The results show an improvement in the chatbot’s understanding and performance increase to most forms of unconstrained languages available on social media. The BLEU score is used to evaluate the efficiency before and after the normalization of sentences. Results show that the microtext normalization technique promises to increase unconstrained text understanding in a pre-trained chatbot.