Density estimation with LLMs: a geometric investigation of in-context learning trajectories

Toni J.B. Liu, Nicolas Boulle, Raphaël Sarfati, Christopher Earls

Keywords: Large language models, in-context learning, Intensive Principal Component Analysis, density estimation, Mechanistic interpretations, High-dimensional learning dynamics, dimensionality reduction

TL;DR: We visualize and analyze LLMs' trajectories of density estimations from data observed in context.

Abstract:

Large language models (LLMs) demonstrate remarkable emergent abilities to perform in-context learning across various tasks, including time series forecasting.
This work investigates LLMs' ability to estimate probability density functions (PDFs) from data observed in-context;
such density estimation (DE) is a fundamental task underlying many probabilistic modeling problems.
We leverage the Intensive Principal Component Analysis (InPCA) to visualize and analyze the in-context learning dynamics of LLaMA-2 models.
Our main finding is that these LLMs all follow similar learning trajectories in a low-dimensional InPCA space, which are distinct from those of traditional density estimation methods like histograms and Gaussian kernel density estimation (KDE).
We interpret the LLaMA in-context DE process as a KDE with an adaptive kernel width and shape.
This custom kernel model captures a significant portion of LLaMA's behavior despite having only two parameters.
We further speculate on why LLaMA's kernel width and shape differs from classical algorithms, providing insights into the mechanism of in-context probabilistic reasoning in LLMs.
Our codebase, along with a 3D visualization of an LLM's in-context learning trajectory, is publicly available at https://github.com/AntonioLiu97/LLMICL_inPCA.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9877