Semantic F1 Scores: Fair Evaluation Under Fuzzy Class Boundaries

TMLR Paper8107 Authors

Abstract:

We propose Semantic F1 Scores, novel evaluation metrics for subjective or fuzzy multi-label classification that quantify semantic relatedness between predicted and gold labels. Unlike the conventional F1 metrics that treat semantically related predictions as complete failures, Semantic F1 incorporates a label similarity matrix to compute soft precision-like and recall-like scores, from which the Semantic F1 scores are derived. Unlike existing similarity-based metrics, our novel two-step precision-recall formulation enables the comparison of label sets of arbitrary sizes without discarding labels or forcing matches between dissimilar labels. When the similarity matrix reflects meaningful semantic or downstream relationships between labels, by granting partial credit for semantically related but nonidentical labels, Semantic F1 better reflects the realities of domains marked by human disagreement or fuzzy category boundaries.
Through theoretical justification and extensive empirical validation on synthetic and real data, we show that Semantic F1 demonstrates greater interpretability and ecological validity.
Semantic F1 is applicable in domains where practitioners can specify, validate, or inherit a domain-appropriate similarity matrix. We show that the metric is robust under moderate misspecification, and emphasize that invalid similarity matrices can make any similarity-based evaluation misleading.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Jingyan_Wang1

Submission Number: 8107