Do Llamas See the Periodic Table in 3D? Geometry and Layerwise Representations

Ge Lei, Samuel J. Cooper

Track: Track 1: Original Research/Position/Education/Attention Track

Keywords: Knowledge representation, Intervention, Mechanistic interpretability

Abstract: Large Language Models (LLMs) show impressive capacity to synthesize scientific knowledge but struggle with basic arithmetic, raising concerns about reliability. As materials science increasingly leverages LLMs for hypothesis generation, it is essential to understand how they encode specialized knowledge. Here, we investigate how the open-source Llama series of LLMs represent the periodic table of elements. We identify a 3D spiral structure in the hidden states of LLMs that aligns with the conceptual structure of the periodic table, suggesting that LLMs can reflect the geometric organization of scientific concepts learned from text. Linear probing reveals that middle layers encode continuous, overlapping attributes that enable indirect recall, while deeper layers sharpen categorical distinctions and incorporate linguistic context. These findings suggest that LLMs represent symbolic knowledge not as isolated facts, but as structured geometric manifolds that intertwine semantic information across layers. We hope this inspires further exploration into the interpretability mechanisms of LLMs within chemistry and materials science, enhancing trust of model reliability, guiding model optimization and tool design, and promoting mutual innovation between science and AI.

Submission Number: 116