Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play

ICLR 2026 Conference Submission12636 Authors

Keywords: Speech Role-play, Role-play Evaluation, Speech Evaluation Models

TL;DR: Speech-DRAME introduces a bilingual benchmark for speech role-play, combining Archetype (top-down) and Realism (bottom-up) evaluation, aligning speech evaluation models with human judgments to outperform zero-shot audio LLM judges.

Abstract:

Role-play has become a key testbed for generative models, expanding from text-only dialogue to multimodal interaction. Extending role-play to speech captures prosody, emotion, and delivery, but also poses new evaluation challenges. Current pipelines often use audio large language models (ALLMs) as zero-shot judges, which miss paralinguistic cues, collapse multiple aspects into coarse scores, and rely on synthetic speech references that fail to reflect real-world roles. We present Speech-DRAME, a unified framework that contributes at three levels: (i) Speech-DRAME-EvalBench, an evaluation benchmark with bilingual human-annotated data and protocols for training and testing speech evaluation models (SEMs), (ii) DRAME-Eval, a fine-tuned evaluation model that substantially outperforms zero-shot and few-shot ALLMs, and (iii) Speech-DRAME-RoleBench, a speech role-play benchmark that leverages DRAME-Eval as an automatic judge to compare speech foundation models (SFMs). Speech-DRAME distinguishes between two complementary evaluation strategies: Archetype Evaluation, a top-down approach measuring adherence to broad role archetypes, and Realism Evaluation, a bottom-up approach grounded in real human speech that emphasizes nuanced role quality. Compared to zero-shot ALLM judges, DRAME-Eval achieves stronger agreement with human ratings (Pearson correlation improves from 0.480 to 0.629 in archetypes, and from 0.390 to 0.625 in realism).
By integrating transparent benchmark resources, modeling approaches, and system-level evaluation, Speech-DRAME provides the first comprehensive, reproducible foundation for assessing spoken role-play.

Primary Area: datasets and benchmarks

Submission Number: 12636