SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training

Published: 28 Oct 2023, Last Modified: 15 Dec 2023NeurIPS2023-AI4Science PosterEveryoneRevisionsBibTeX
Keywords: Symbolic Mathematics, Pre-training, Transformers, Deep Learning
TL;DR: Introducing a multi-modal pre-training approach that bridges the understanding between symbolic math expressions and their numeric data observations
Abstract: In scientific inquiry, symbolic mathematical equations play a fundamental role in modeling complex natural phenomena. Leveraging the power of deep learning, we introduce SNIP, a Multi-Modal Symbolic-Numeric Pre-training framework. By employing joint contrastive learning between symbolic and numeric domains, SNIP enhances their mutual alignment in pre-trained embeddings. Latent space analysis reveals that symbolic supervision significantly enriches the embeddings of numeric data, and vice versa. Evaluations across diverse tasks, including symbolic-to-numeric and numeric-to-symbolic property prediction, demonstrate SNIP's superior performance over fully supervised baselines. This advantage is particularly pronounced in few-shot learning scenarios, making SNIP a valuable asset in situations with limited available data.
Submission Track: Original Research
Submission Number: 130