An Operational Deep Learning Pipeline for Classifying Life Events from Individual Tweets

Xinsong Du, Jiang Bian, Mattia Prosperi

Abstract: We here present an operational deep learning pipeline for classifying life events from individual tweets, using job loss as a use case and Twitter data collected between 2010 and 2013 (historic sample from the public stream). The pipeline includes identification of keywords through snowball sampling, multiple rater manual annotation, supervised deep learning, text processing (word embedding, bag of words) and architecture selection (convolutional, shallow-and-wide convolutional, and long-short-term memory) with parameter optimization, external validation and feedback learning. After model optimization, a shallow-and-wide network with a pre-trained 200-dimensional word2vec achieved a precision of 78% (over an average single keyword precision of 50%) and an area under receiver operating characteristic of 86%. Precision and recall also increased by 5% using bag of words. When tested on tweets with ambiguous annotations (i.e. tweets that were hard for human annotators to classify), the network achieved 65% precision. Finally, on a random set of tweets that did not contain any of the snowballed keywords, 30% were classified as job loss events; this putatively false positive set can be used to reinforce the learner’s training. In conclusion, the pipeline streamlines both the manual and automated process, providing feedback reinforcement (snowballing and external tweets), and shows good performance on classifying individual tweets on the use case, potentially saving human resources needed to collate such data for research studies.