Contrastive Classification via Linear Layer Extrapolation
Abstract: Early-exiting predictions in a deep Transformer network evolve from layer to layer in a somewhat smooth process. This has been exploited in language modeling to improve factuality (Chuang et al., 2023), with the observation that factual associations emerge in later layers. We find a similar process multiway emotion classification, motivating Linear Layer Extrapolation, which finds stable improvements by recasting contrastive inference as linear extrapolation. Experiments across multiple models and emotion classification datasets find that Linear Layer Extrapolation outperforms standard classification on fine-grained emotion analysis tasks.
Paper Type: Short
Research Area: Sentiment Analysis, Stylistic Analysis, and Argument Mining
Research Area Keywords: applications, sentiment analysis, language modeling
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models
Languages Studied: English
Submission Number: 3832
Loading