Position: Beyond Euclidean – Foundation Models Should Embrace Non-Euclidean Geometries

Neil He, Jiahong Liu, Buze Zhang, Ngoc Bui, Ali Maatouk, Irwin King, Menglin Yang, Melanie Weber, Rex Ying

Keywords: Foundation Models, Non-Euclidean Space, Pretraining, Distortion, Hyperbolic Space

Abstract: In the era of foundation models and Large Language Models (LLMs), Euclidean space has been the de facto geometric setting for machine learning architectures. However, recent literature has demonstrated that this choice comes with fundamental limitations. At a large scale, real-world data often exhibits inherently non-Euclidean structures, such as multi-way relationships, hierarchies, symmetries, and non-isotropic scaling, in a variety of domains, such as languages, vision, and the natural sciences. It is challenging to effectively capture these structures within the constraints of Euclidean spaces. This position paper argues that moving beyond Euclidean geometry is not merely an optional enhancement but a necessity to maintain the scaling law for the next-generation of foundation models. By adopting these geometries, foundation models could more efficiently leverage the aforementioned structures. Task-aware adaptability that dynamically reconfigures embeddings to match the geometry of downstream applications could further enhance efficiency and expressivity. Our position is supported by a series of theoretical and empirical investigations of prevalent foundation models. Finally, we outline a roadmap for integrating non-Euclidean geometries into foundation models, including strategies for building geometric foundation models via fine-tuning, training from scratch, and hybrid approaches.

Neurips Id: 623

Neurips Title: Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries

Neurips Authors: Neil He, Jiahong Liu, Buze Zhang, Ngoc Bui, Ali Maatouk, Menglin Yang, Irwin King, Melanie Weber, Rex Ying

Confirmation: By submitting, authors affirm that all information provided is complete, truthful, and anonymized, including the NeurIPS-25 paper ID, author list, title, original PDF, reviews, meta-review, and author responses. The organizing committee reserves the right to verify the accuracy of all submitted materials after acceptance, in coordination from the NeurIPS-25 Organizing Committee. Providing false or misleading information constitutes a violation of scientific integrity. Submissions found to violate this policy will be desk-rejected.

Submission Number: 6