Don’t Just Pay Attention, PLANT It: Transfer L2R Models to Fine-tune Attention in Extreme Multi-Label Text Classification For ICD Coding
Abstract: The keystone of state-of-the-art Extreme Multi-Label Text Classification (XMTC) models is the multi-label attention layer within the decoder, which deftly directs label-specific focus to salient tokens in input text. Nonetheless, the process of acquiring these optimal attention weights is onerous and resource-intensive. To alleviate this strain, we introduce PLANT \--- Pretrained and Leveraged AtteNTion \--- an innovative transfer learning strategy to fine-tune XMTC decoders. The central notion involves transferring a pretrained learning-to-rank (L2R) model, utilizing its activations as attention weights, thereby serving as the planted attention layer in the decoder.
On the full MIMIC-III dataset, \plant excels in four out of seven metrics and surpasses in five for the top-50 code set, demonstrating its effectiveness. Remarkably, for the rare-50 code set, \plant achieves a significant $12.7-52.2\%$ improvement in four metrics. On MIMIC-IV, it leads in three metrics.
Notably, in low-shot scenarios, \plant matches traditional attention models' precision despite using significantly less data ($\frac{1}{10}$ for precision at $5$, $\frac{1}{5}$ for precision at $15$), highlighting its efficiency with skewed label distributions.
Paper Type: Long
Research Area: Machine Learning for NLP
Research Area Keywords: Efficient/Low-Resource Methods for NLP, Information Extraction, Information Retrieval and Text Mining, Language Modeling, Interpretability and Analysis of Models for NLP
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings
Languages Studied: English
Submission Number: 426
Loading