Beyond Testers’ Biases: Guiding Model Testing with Knowledge Bases using LLMs

Chenyang Yang, Rishabh Rustogi, Rachel Brower-Sinning, Grace Lewis, Christian Kaestner, Tongshuang Wu

Submission Type: Regular Long Paper

Submission Track: Human-Centered NLP

Submission Track 2: Interpretability, Interactivity, and Analysis of Models for NLP

Keywords: model testing, knowledge base, large language models, human-centered NLP

Abstract:

Current model testing work has mostly focused on creating test cases.
Identifying what to test is a step that is largely ignored and poorly supported.
We propose Weaver, an interactive tool that supports requirements elicitation for guiding model testing.
Weaver uses large language models to generate knowledge bases and recommends concepts from them interactively, allowing testers to elicit requirements for further testing.
Weaver provides rich external knowledge to testers and encourages testers to systematically explore diverse concepts beyond their own biases.
In a user study, we show that both NLP experts and non-experts identified more, as well as more diverse concepts worth testing when using Weaver.
Collectively, they found more than 200 failing test cases for stance detection with zero-shot ChatGPT.
Our case studies further show that Weaver can help practitioners test models in real-world settings,
where developers define more nuanced application scenarios (e.g., code understanding and transcript summarization) using LLMs.

Submission Number: 2034