[Short] RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics

Zhengyang Qi, Charles Andrew Dickens, Derek Pham, Amanda Dsouza, Armin Parchami, Frederic Sala, Paroma Varma

Keywords: Large Language Models, LLM Evaluation and Benchmarking, Rubrics Based Evaluation

Abstract:

Rubric-based evaluation is widely used in LLM benchmarks and training pipelines for open-ended, less verifiable tasks.
While prior work has demonstrated the effectiveness of rubrics using downstream signals such as reinforcement learning outcomes, there remains no principled way to diagnose rubric quality issues from such aggregated or downstream signals alone.
To address this gap, we introduce $\textbf{RIFT: RubrIc Failure mode Taxonomy}$, a taxonomy for systematically characterizing failure modes in rubric composition and design.
RIFT consists of eight failure modes organized into three high-level categories: $\textit{Reliability Failures}$, $\textit{Content Validity Failures}$, and $\textit{Consequential Validity Failures}$.
RIFT is developed using grounded theory by iteratively annotating rubrics drawn from five diverse benchmarks spanning general instruction following, code generation, creative writing, and expert-level deep research, until no new failure modes are identified.
We evaluate the consistency of the taxonomy by measuring agreement among independent human annotators, observing fair agreement overall ($\textbf{87\% pairwise agreement}$ and $\textbf{0.64 average Cohen’s kappa}$).
Finally, to support scalable diagnosis, we propose automated rubric quality metrics and show that they align with human failure-mode annotations, achieving up to $0.86$ F1.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 143