Objective Matters: Fine-Tuning Objectives Shape Safety, Robustness, and Persona Drift

ACL ARR 2026 January Submission5457 Authors

Keywords: Large language models, Fine-tuning, Adversarial robustness, Model alignment, Safety–capability tradeoffs

Abstract: Fine-tuning on benign data can still degrade alignment and adversarial robustness, yet direct analysis of the role of fine-tuning objectives in shaping these safety outcomes remain limited. We present a controlled comparison of six fine-tuning objectives---Supervised Fine-Tuning, Direct Preference Optimization, Conditional Fine-Tuning, Inoculation Prompting, Odds Ratio Preference Optimization, and KL-regularized fine-tuning---holding data, domain, architecture, and optimization fixed. Across closed-form reasoning and open-ended generation tasks, we find that objective choice induces systematic, scale-dependent shifts along the safety–capability frontier. At small training budgets, robustness is similar across objectives but capability differs. At larger budgets, objectives diverge sharply: supervised and preference-based tuning tightly couple capability gains to increased adversarial vulnerability and persona drift, while objectives that constrain learning signals--especially ORPO and KL-regularization---substantially mitigate both. Fine-tuning objectives therefore matter little for safety at small scales but become a primary driver of adversarial robustness and latent persona stability as training scale increases.

Paper Type: Long

Research Area: Safety and Alignment in LLMs

Research Area Keywords: adversarial attacks, fine-tuning, robustness, safety and alignment, evaluation, data influence

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 5457