A Compute-Matched Study of Hidden Layer Distillation for LLM Pre-Training

Maxime Guigon, Lucas Dixon, Michael Eli Sander

Keywords: Knowledge Distillation; Compute-Optimal Training; Hidden Layer Distillation; LLM Pre-training; FLOPs-Matched Evaluation

TL;DR: We present a strict, compute-matched evaluation of Hidden Layer Distillation (HLD) for pre-training decoder-only LLMs. While HLD yields a modest perplexity improvement over standard distillation, this gain does not transfer to downstream tasks.

Abstract: Knowledge Distillation (KD) is a critical tool for training Large Language Models (LLMs), yet the majority of research focuses on approaches that rely solely on output logits, neglecting semantic information in the teacher's intermediate representations. While Hidden Layer Distillation (HLD) showed potential for encoder architectures, its application to decoder-only pre-training at scale remains largely unexplored. Through compute-controlled experiments, we benchmark HLD against logit-based KD and self-supervised baselines with Gemma3 3.4B as teacher and 123M and 735M students trained on up to 168B tokens from the C4 dataset. Our experiments show that HLD does not consistently outperform standard KD on downstream evaluation tasks. Nevertheless, we show that HLD can yield a systematic perplexity gain over KD across all shared-hyperparameter configurations, suggesting that a latent signal can be extracted, but a breakthrough may be needed for it to play a more significant role in LLM pre-training.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 51