SPONGE: Competing Sparse Language Representations for Effective Knowledge Transfer

TMLR Paper4971 Authors

Abstract:

In domains with privacy constraints, most knowledge resides in siloed datasets, hindering the development of a model with all relevant knowledge for a task.
Clinical NLP is a prime example of these constraints in practice.
Research in this area typically falls back to the canonical setting of sequential transfer learning, where a model pre-trained on large corpora is finetuned on a smaller annotated dataset.
An avenue for knowledge transfer among diverse clinics is multi-step sequential transfer learning since models are more likely to be shared than private clinical data.
This setting poses challenges of cross-linguality, domain diversity, and varying label distributions which undermine generalisation.
We propose SPONGE, an efficient prototypical architecture that leverages competing sparse language representations.
These encompass distributed knowledge and create the necessary level of redundancy for effective transfer learning across multiple datasets.
We identify that prototypical classifiers are critically sensitive to label-recency bias which we mitigate with a novel strategy at inference time. SPONGE in combination with this strategy significantly boosts generalisation performance to unseen data.
With the help of medical professionals, we show that the explainability of our models is clinically relevant.
We make all source code available.

Submission Length: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=GDlo4WB0lj

Changes Since Last Submission:

We found that the template we used imported a different font, as clearly pointed out by the editors.
Thus, we removed those imports and additionally adapted a few table captions.
For a better presentation of our results, we also opted for consolidating four tables into two.
Thank you for your patience since this was hard to debug.

Assigned Action Editor: ~changjian_shui1

Submission Number: 4971