Conformal Tail Risk Control for Large Language Model Alignment

Catherine Chen, Jingyan Shen, Zhun Deng, Lihua Lei

TL;DR: We develop a method that leverages conformal risk control and asymptotics of L-statistics to achieve guaranteed tail risk control for any given LLM under no assumptions on the statistical properties and the alignment quality of the LLM.

Abstract: Recent developments in large language models (LLMs) have led to their widespread usage for various tasks. The prevalence of LLMs in society implores the assurance on the reliability of their performance. In particular, risk-sensitive applications demand meticulous attention to unexpectedly poor outcomes, i.e., tail events, for instance, toxic answers, humiliating language, and offensive outputs. Due to the costly nature of acquiring human annotations, general-purpose scoring models have been created to automate the process of quantifying these tail events. This phenomenon introduces potential human-machine misalignment between the respective scoring mechanisms. In this work, we present a lightweight calibration framework for blackbox models that ensures the alignment of humans and machines with provable guarantees. Our framework provides a rigorous approach to controlling any distortion risk measure that is characterized by a weighted average of quantiles of the loss incurred by the LLM with high confidence. The theoretical foundation of our method relies on the connection between conformal risk control and a traditional family of statistics, i.e., L-statistics. To demonstrate the utility of our framework, we conduct comprehensive experiments that address the issue of human-machine misalignment.

Lay Summary: Recent developments in large language models (LLMs) have led to their widespread usage for various tasks. The prevalence of LLMs in society implores the assurance on the reliability of their performance. In particular, risk-sensitive applications demand meticulous attention to unexpectedly poor outcomes, i.e., tail events, for instance, toxic answers, humiliating language, and offensive outputs. Due to the costly nature of acquiring human annotations, machine scoring models have been created to automate the process of quantifying these tail events. This phenomenon introduces potential human-machine misalignment between the respective scoring mechanisms. In this work, we present a lightweight framework that ensures the alignment of humans and machines with provable guarantees. Our method provides a rigorous approach to controlling any tail risk incurred by the LLM with high confidence. The theoretical foundation of our method relies on the connection between conformal risk control and a traditional family of statistics. This approach not only ensures the alignment of human and machine scores, but also the safety of LLM outputs.

Link To Code: https://github.com/jy-evangeline/DRC

Primary Area: Theory

Keywords: large language models, conformal risk control, human-machine alignment, model calibration, L-statistics

Submission Number: 14637