Language Models Do Not Embed Numbers Continuously

Alex Owen Davies, Roussel Desmond Nzoyem, Nirav Ajmeri, Telmo M Silva Filho

Track: Regular Track (Page limit: 6-8 pages)

Keywords: Numerical Reasoning, Representation Learning, Embedding Quality, PCA, RAG

TL;DR: Popular embedding models fail to preserve the simple one-dimensional nature of numbers, explaining why they struggle with numerical reasoning in scientific and mathematical applications.

Abstract:

Recent research has extensively studied how large language models manipulate integers in specific arithmetic tasks, and on a more fundamental level, how they represent numeric values.
These previous works have found that language model embeddings can be used to reconstruct the original values, however, they do not evaluate whether language models actually model continuous values as continuous.
Using expected properties of the embedding space, including linear reconstruction and principal component analysis, we show that language models not only represent numeric spaces as non-continuous but also introduce significant noise.
Using models from three major providers (OpenAI, Google Gemini and Voyage AI), we show that while reconstruction is possible with high fidelity ($R^2 \geq 0.95$), principal components only explain a minor share of variation within the embedding space.
This indicates that many components within the embedding space are orthogonal to the simple numeric input space.
Further, both linear reconstruction and explained variance suffer with increasing decimal precision, despite the ordinal nature of the input space being fundamentally unchanged.
The findings of this work therefore have implications for the many areas where embedding models are used, in-particular where high numerical precision, large magnitudes or mixed-sign values are common.

Submission Number: 4