A teacher-teacher framework for clinical language representation learning

Feiqing Huang, Shenghan Zhang, Sara Morini Sweet, Tianxi Cai

Keywords: clinical language models, teacher-teacher framework, knowledge alignment

TL;DR: This paper introduces a teacher-teacher paradigm where two pretrained LLMs achieve knowledge exchange and alignment through a two-step, few-epoch training of the LINE module with a well-designed alignment objective.

Abstract:

In recent years, there has been a proliferation of ready-to-use large language models (LLMs) designed for various applications, both general-purpose and domain-specific. Instead of advocating for the development of a new model or continuous pretraining of an existing one, this paper introduces a pragmatic teacher-teacher framework to facilitate mutual learning between two pre-existing models.
By leveraging two teacher models possessing complementary knowledge, we introduce a LIghtweight kNowledge alignmEnt (LINE) module aimed at harmonizing their knowledge within a unified representation space. This framework is particularly valuable in clinical settings, where stringent regulations and privacy considerations dictate the handling of detailed clinical notes. Our trained LINE module excels in capturing critical information from clinical notes, leveraging highly de-identified data. Validation and downstream tasks further demonstrate the effectiveness of the proposed framework.

Primary Area: Natural language processing

Submission Number: 20239