ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment

Charlie Masters, Stefano V. Albrecht

Keywords: AI alignment, reward models, preference learning, verifiable rubrics, multi-agent game

TL;DR: Framing Reward Modelling for MAS as a collaboration between a manager and a stakeholder to generate natural language rubrics to align workers.

Abstract:

As agents based on large language models are increasingly deployed to long-horizon tasks, maintaining their alignment with stakeholder preferences becomes critical. Effective alignment in such settings requires reward models that are interpretable so that stakeholders can understand and audit model objectives. Moreover, reward models must be capable of steering agents at interaction time, allowing preference shifts to be incorporated without retraining.
We introduce ARCANE, a framework that frames alignment as a multi-agent collaboration problem that dynamically represents stakeholder preferences as natural-language rubrics: weighted sets of verifiable criteria that can be generated on-the-fly from task context.
Inspired by utility theory, we formulate rubric learning as a reconstruction problem and develop a regularized Group-Sequence Policy Optimization (GSPO) procedure that balances interpretability, faithfulness, and computational efficiency.
Using a corpus of 219 labeled rubrics derived from the GDPVal benchmark, we evaluate ARCANE on challenging professional tasks requiring multi-step reasoning and tool use. Learned rubrics produce compact, legible evaluations and enable configurable trade-offs (e.g., correctness vs. conciseness) without retraining. Together, these results suggest that rubric-based reward models offer a promising path toward interpretable, test-time adaptive alignment for complex, long-horizon AI systems.

Submission Number: 31