Early Exit: A Nature Capability in Transformer-based Model Unveiling By joint Optimization

Anonymous

Abstract: Large language models demonstrate remarkable performance across various downstream tasks but face challenges with slow inference speeds due to their extensive parameters. The early exit approach, involving obtaining results from different intermediate layers for each token before reaching the final layer, offers a promising solution for accelerating auto-regressive decoding. However, additional output layers and the complexities of joint optimization challenges hinder the implementation of early exit in large language models (LLMs).In this paper, we explore the possibility of early exit within LLMs using the original final output layer and without relying on joint optimization. Our findings indicate that early exit is a natural capability within Transformer-based models. While joint optimization is not strictly necessary for early exit capability, it must be employed to address challenges by 1) improving the accuracy of locating the optimal early exit layer through gating functions and 2) mitigating key-value copy issues. Additionally, we uncover tendencies in early exit behavior that make the model struggle to predict the first sub-word in a word and identify the potential for early exit strategies based on sub-layers.

Paper Type: long

Research Area: Generation

Contribution Types: Model analysis & interpretability

Languages Studied: English