Hierarchical Analysis: Monotonicity of Layerwise performance in Large Language Models

Jianfa Lai, Weihao Lu, Dongming Huang, Hehu Xie, Jie Luo, Qian Lin

Keywords: Monotonicity, Layerwise Performance, Large Language Models (LLMs), Hierarchical Analysis

TL;DR: The paper introduces a framework to analyze the "monotonicity" phenomenon in LLMs, showing how their performance consistently improves across layers, which can enhance inference efficiency for various tasks.

Abstract:

We introduce a quantitative framework to evaluate how Large Language Models (LLMs) learn tasks across all layers, revealing a `monotonicity phenomenon'. Specifically:
i) performance at each layer consistently improves from one layer to the next on the pre-training set, and
ii) this improvement is consistently observed across various downstream tasks. This monotonicity phenomenon indicates that LLMs effectively capture complex hierarchical features across diverse datasets. For example, our study on the abstraction of concepts using linear representations in word embeddings shows that the clarity of these abstractions progressively increases with each layer.
Finally, by leveraging this monotonicity, we can significantly reduce inference time and memory requirements by selecting the most appropriate layer, thereby enhancing the efficiency of LLMs in real-world applications.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10142