Self-Supervised Feature Re-Representation via Lennard-Jones Potential Loss

Jianlong Kwan, Jiayu Xiong, Dilong Li, Jing Wang

Keywords: Physics-Inspired Optimization, Pluggable Self-Supervised Loss, Lennard-Jones Potential

TL;DR: A Loss function via Lennard-Jones potential for generalized gradient optimization.

Abstract:

The Lennard-Jones potential, initially developed to model molecular interactions, is characterized by a repulsive force at short distances to prevent over-clustering and an attractive force at longer distances to maintain balanced proximity, resembling the equilibrium-seeking behavior of particles in natural systems. This offers a potential pathway for more orderly entropy reduction in higher-order features.
This paper introduces a self-supervised approach for feature re-representation, utilizing a Lennard-Jones potential loss to constrain the gradient directions between positive and negative features in computer vision tasks. Unlike supervised learning directly driven by downstream tasks or contrastive learning with multi-label data pairs and multi-feature extractors, the proposed loss term integrates with existing task-specific losses by directly constraining gradient directions, thereby enhancing the feature learning process.
Extensive theoretical analysis and experimental results demonstrate that, across various domains, datasets, network architectures, and tasks, models incorporating the Lennard-Jones potential loss significantly outperform baseline models without this auxiliary loss in both accuracy and robustness. This approach highlights the potential of physics-inspired loss functions to improve deep learning optimization.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1046