EVALUATING WORD REPRESENTATIONS IN A TECHNICAL LANGUAGE PROCESSING PIPELINE

Ajay Varma Nandyala, Sarah Lukens, Sundararam Rathod, Pratiksha Agrawal

Abstract:

The recent explosion of advancements in natural language
processing (NLP) are encouraging in the industrial sector for
leveraging the volumes of unstructured, technical data that
currently sit unused. However, results from direct
application of many NLP pipelines to technical text often fail
to address the business needs of industrial companies. One
requirement for satisfactory performance is an effective
representation of the unstructured text in a form which
contains the information required for an application task. We
know of no standard methodology for evaluating word
representations for technical text tailored to industry
needs. In this paper, we propose guidance and methods for
evaluating the performance of word representations for
industrial use-cases.