A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory

Abstract:

Standard LLM evaluation practices compress diverse abilities into single scores, obscuring their inherently multidimensional nature.
We present *JE-IRT*, a geometric item-response framework that embeds both LLMs and questions in a shared space.
For question embeddings, the **direction** encodes semantics and the **norm** encodes difficulty, while correctness on each question is determined by the geometric interaction between the model and question embeddings.
This geometry replaces a global ranking of LLMs with topical specialization and enables smooth variation across related questions.
Building on this framework, our experimental results reveal that out-of-distribution behavior can be explained through directional alignment, and that larger norms consistently indicate harder questions.
Moreover, JE-IRT naturally supports generalization: once the space is learned, new LLMs are added by fitting a single embedding.
The learned space further reveals an LLM-internal taxonomy that only partially aligns with human-defined subject categories.
We also show that simple linear probes of the embedding space recover cross-subject ability directions, such as an arithmetic axis that highlights quantitatively demanding questions in seemingly distant subjects like **virology** and **global facts**.
JE-IRT thus establishes a unified and interpretable geometric lens that connects LLM abilities with the structure of questions, offering a distinctive perspective on model evaluation and generalization.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Ran_Tian1

Submission Number: 7474