Scaling Policy Compliance Assessment in Language Models using Policy Reasoning Traces

Joseph Marvin Imperial, Harish Tayyar Madabushi

Keywords: policy compliance, reasoning trace, chain-of-thought, large language model

TL;DR: We propose Policy Reasoning Traces, a form of pseudo-expert reasoning imitation derived from frontier reasoning models to improve any off-the-shelf model's policy compliance capabilities.

Abstract: Policy compliance assessment is a fundamental task of evaluating whether an input case strictly complies with a set of human-defined rules, more generally known as *policies*. In practice, human experts follow a systematic, step-by-step process to identify violations with respect to specific stipulations outlined in the policy. However, such documentation of gold-standard, expert-level reasoning processes is costly to acquire. In this paper, we introduce Policy Reasoning Traces (PRT), a form of specialized generated reasoning chains that serve as a *reasoning bridge* to improve an LLM's policy compliance assessment capabilities. Our empirical evaluations demonstrate that the use of PRTs for both inference-time and training-time scenarios significantly enhances the performance of open-weight and commercial models, setting a new state-of-the-art for HIPAA and GDPR policies. Beyond accuracy gains, we also highlight how PRTs can improve an LLM's ability to accurately cite policy clauses, as well as influence compliance decisions through their high utilization from the raw chains-of-thought.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 9485