Semantic Search in Olympiad Math Problems

Erbol Esengulov, Filip Ilievski

Track: Type D (Master/Bachelor Thesis Abstracts)

Keywords: Math, AI, Search engine, Semantical AI, NLP, Clustering, Ranking, Embedding Models, Mathematical Retrieval, Similarity

Abstract: Identifying similar mathematical problems based on reasoning structure rather than surface features remains a key challenge in educational search systems. This thesis explores the feasibility of using semantic search—powered by pretrained transformer models—to cluster and retrieve Olympiad-level number theory problems based on their solution strategies. We construct three custom benchmarks from a widely used textbook: a section-labeled dataset for clustering, a strategy-labeled dataset with shorter text segments, and a human-annotated ranking dataset comparing problems by their semantic proximity. Several pretrained language models, including MathBERT, SciBERT, and Longformer, are evaluated using clustering and ranking metrics. Results show that long-context models like Longformer outperform domain-specific models, especially in ranking tasks, suggesting the importance of larger attention windows for capturing mathematical strategy. However, section-based labels in textbooks are found to be unreliable for fine-tuning, and summarization-based approaches degrade model performance. Fine-tuning with triplet loss yielded limited improvements. Our findings highlight the limitations of existing datasets, the need for human-curated benchmarks, and the importance of architectural choices when modeling mathematical semantics. Future work should explore hybrid symbolic-neural models and improved methods for long text representation.

Serve As Reviewer: ~Filip_Ilievski1

Submission Number: 38