From Implemented to Expected Behaviors: Leveraging Regression Oracles for Non-regression Fault Detection using LLMs

Stefano Ruberto, Judith Perera, Gunel Jahangirova, Valerio Terragni

Abstract: Automated test generation tools often produce assertions that reflect implemented behavior, limiting their usage to regression testing. In this paper, we propose LLMProphet, a black-box approach that applies Few-Shot Learning with LLMs, using automatically generated regression tests as context to identify non-regression faults without relying on source code. By employing iterative cross-validation and a leave-one-out strategy, LLMProphet identifies regression assertions that are misaligned with expected behaviors. We outline LLMProphet’s workflow, feasibility, and preliminary findings, demonstrating its potential for LLM-driven fault detection.