Towards a framework for reliable performance evaluation in defect prediction

Xutong Liu, Shiran Liu, Zhaoqiang Guo, Peng Zhang, Yibiao Yang, Huihui Liu, Hongmin Lu, Yanhui Li, Lin Chen, Yuming Zhou

Abstract: Highlights•Introducing an evaluation framework MATTER to address the challenge of evaluating performance in software defect prediction (SDP) models.•Surprising findings using MATTER: Recent representative SDP models did not significantly outperform the simple baseline model ONE.•Supportive feedback from researchers and practitioners: A survey on whether MATTER aligns with developers' preferences for SDP models.•Urge adoption of MATTER for assessing new SDP models' usefulness, promoting reliable scientific progress in defect prediction.