Customized meta-dataset for automatic classifier accuracy evaluation

Yan Huang, Zhang Zhang, Yan Huang, Qiang Wu, Han Huang, Yi Zhong, Liang Wang

Abstract: Highlights•Two authors named Yan Huang in pinyin, one PostDoc (1st author), the other an Associate Prof (3rd author).•Paper on ACAEval for classifier accuracy. Uses meta-dataset technique for unlabeled real-world data.•Customized meta-dataset for ACAEval to address distribution shift between samples and test set.•Our sample set considers label flip issue. Introduces random indicator and FD margin loss.•Experiments demonstrate our ACAEval matches existing methods and surpasses dataset-level regression.