Equitable Elicitation

Reid McIlroy-Young, Lunjia Hu, Han Shao, Linjun Zhang, Cynthia Dwork

Keywords: Fairness, Multi-accuracy, simulated participants, Reinforcement Learning, Signal Elicitation

TL;DR: We build an AI to address the problem of providing accurate and equitable elicitation of skills

Abstract:

Individuals with similar qualifications and skills may vary in their demeanor, or outward manner: some tend toward self-promotion while others are more self-effacing. Comparing the self-descriptions of equally qualified job-seekers with different self-presentation styles is therefore problematic. We build an interactive AI for _skill elicitation_ that provides accurate determination of skills _while simultaneously allowing individuals to speak in their own voice._ Such a system can be used, for example, when a new user joins a professional networking platform, or when seeking to match employees to needs during a company reorganization.

We articulate the equitable elicitation problem. We build an interactive AI to address the problem, providing accurate and equitable determination of skills. To this end, we train an LLM to act as synthetic humans to provide training data, and we provably enforce a mathematically rigorous notion of equitability, ensuring that the covariance between self-presentation manner and skill evaluation error is small.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 23167