Revisiting Intermediate-Layer Matching in Knowledge Distillation: Layer-Selection Strategy Doesn’t Matter (Much)

ACL ARR 2025 February Submission2693 Authors

Abstract: Knowledge distillation (KD) is a popular method of transferring knowledge from a large "teacher" model to a small "student" model. KD can be divided into two categories: prediction matching and intermediate-layer matching. We explore an intriguing phenomenon: layer-selection strategy does not matter (much) in intermediate-layer matching. In this paper, we show that seemingly nonsensical matching strategies such as matching the teacher's layers in reverse still result in surprisingly good student performance. We provide an interpretation for this phenomenon by examining the angles between teacher layers viewed from the student's perspective.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: Knowledge Distillation

Contribution Types: Model analysis & interpretability

Languages Studied: English, English to Romanian Translation

Submission Number: 2693