Preference Learning from Physics-Based Feedback: Tuning Language Models to Design BCC/B2 Superalloys

Satanu Ghosh, Collin Holgate, Neal R Brodnik, Doug Downey, Samantha Daly, Tresa Pollock, Samuel Carton

Keywords: Language models, preference tuning, direct preference optimization, alloys

TL;DR: We use DPO to train local language models to generate ideas for BCC/B2 superalloys

Abstract: We apply preference learning to the task of language model-guided design of novel structural alloys. In contrast to prior work that focuses on generating stable inorganic crystals, our approach targets the synthesizeability of a specific structural class: BCC/B2 superalloys, an underexplored family of materials with potential applications in extreme environments. Using three open-weight models (LLaMA-3.1, Gemma-2, and OLMo-2), we demonstrate that language models can be optimized for multiple design objectives using a single, unified reward signal through Direct Preference Optimization (DPO). Unlike prior approaches that rely on heuristic or human-in-the-loop feedback (costly), our reward signal is derived from thermodynamic phase calculations, offering a scientifically grounded criterion for model tuning. To our knowledge, this is the first demonstration of preference-tuning a language model using physics-grounded feedback for structural alloy design. The resulting framework is general and extensible, providing a path forward for intelligent design-space exploration across a range of physical science domains.

Submission Track: Paper Track (Full Paper)

Submission Category: AI-Guided Design

Institution Location: Durham, New Hampshire, United States

AI4Mat Journal Track: Yes

AI4Mat RLSF: Yes

Submission Number: 103