Graph Neural Networks for Adapting Off-the-shelf General Domain Language Models to Low-Resource Specialised DomainsDownload PDF

Published: 08 Jun 2022, Last Modified: 05 May 2023DLG4NLP 2022 OralReaders: Everyone
Keywords: natural language processing, graph convolutional networks, specialised domains
TL;DR: We use graph convolutional networks to add global information about a specialised domain's vocabulary into a large language model
Abstract: Language models encode linguistic proprieties and are used as input for more specific models. Using their word representations as-is for specialised and low-resource domains might be less efficient. Methods of adapting them exist, but these models often overlook global information about how words, terms, and concepts relate to each other in a corpus due to their strong reliance on attention. We consider that global information can influence the results of the downstream tasks, and combination with contextual information is performed using graph convolution networks or GCN built on vocabulary graphs. By outperforming baselines, we show that this architecture is profitable for domain-specific tasks.
0 Replies