HEP-JEPA: A foundation model for collider physics

Published: 05 Mar 2025, Last Modified: 28 Mar 2025ICLR 2025 Workshop AgenticAI PosterEveryoneRevisionsBibTeXCC BY 4.0
Keywords: foundation model, self-supervised learning, particle physics, high energy physics, jepa
Abstract:

We present a transformer architecture-based foundation model for tasks at high-energy particle colliders such as the Large Hadron Collider. We train the model to classify jets using a self-supervised strategy inspired by the Joint Embedding Predictive Architecture. We use the JetClass dataset containing 100M jets of various known particles to pre-train the model with a data-centric approach --- the model uses a fraction of the jet constituents as the context to predict the embeddings of the unseen target constituents. Our pre-trained model fares well with other datasets for standard classification benchmark tasks. We test our model on two additional downstream tasks: top tagging and differentiating light-quark jets from gluon jets. We also evaluate our model with task-specific metrics and baselines and compare it with state-of-the-art models in high-energy physics. Therefore, this work contributes to the development of scientific foundation models by demonstrating how self-supervised transformer architectures can extract deep insights from high-energy physics data.

Submission Number: 36
Loading

OpenReview is a long-term project to advance science through improved peer review with legal nonprofit status. We gratefully acknowledge the support of the OpenReview Sponsors. © 2025 OpenReview