Counterfactual Justice Benchmark (CJB-100): Evaluating Demographic Drift in LLaMA-Based Legal Decision Support

Gokul Srinath Seetha Ram, Rashmi Elavazhagan

Keywords: counterfactual fairness, equal protection, algorithmic fairness, legal AI, LLM bias, demographic drift, risk assessment, constitutional law, justice system, fairness benchmarks

Paper Type: Short papers / work-in-progress

TL;DR: We introduce CJB-100, a 100-scenario counterfactual fairness benchmark that reveals case-level demographic drift in LLaMA-based legal decision support systems despite appearing fair on aggregate.

Abstract:

The Equal Protection Clause requires that similarly situated individuals receive similar treatment. As large language models enter charging, bail, and sentencing workflows, courts must determine whether these systems satisfy that constitutional guarantee. We present the Counterfactual Justice Benchmark (CJB-100), a 100-scenario testbed designed to evaluate whether legal AI systems treat identical defendants differently when only ethnicity changes. Each scenario is paired with five counterfactual personas—White, Black, Latino, Asian, and Middle Eastern—matched on all legally relevant factors. Four LLaMA-family models (Maverick-17B, Scout-17B, Llama-3.3-70B, Llama-3.3-8B) act as AI attorneys, producing structured risk scores (0–10) and outcome recommendations (0–3), for a total of 2,000 evaluations.

Aggregate ethnicity-averaged risk scores appear neutral (3.55–3.59), yet per-case analysis reveals 2–3 point swings in risk solely due to ethnicity cues—sufficient to alter real-world charging, bail, or sentencing outcomes. We show that such disparities constitute counterfactual violations consistent with causal fairness frameworks and equal protection doctrine. Alignment and model scale reduce—but do not eliminate—demographic drift.

This paper makes four contributions: (1) a rigorously controlled counterfactual fairness benchmark for legal AI; (2) a 2,000-sample empirical evaluation across four LLaMA variants; (3) constitutional findings demonstrating that aggregate parity metrics obscure case-level violations; and (4) guidance for adopting counterfactual, per-case audits as prerequisites for deploying AI-generated risk assessments. CJB-100 provides actionable evidence for courts, prosecutors, and policymakers evaluating legal AI under constitutional scrutiny.

Submission Number: 43