A Case Study of Multi-class Classification with Diversified Precision Recall Requirements for Query Disambiguation

Yingrui Yang, Christopher Miller, Peng Jiang, Azadeh Moghtaderi

Abstract: We introduce a new metric for measuring the performance of multi-class classifiers. This metric is a generalization of the f1 score that is defined on binary classifiers, and offers significant improvement over other generalizations such as micro- and macro-averaging. In particular, one can select coefficients that weight the per-class precision and recall, as well as the overall class importance, with a robust mathematical interpretation. When certain parameters are selected our metric yields macro-averaged statistic as a special case. We demonstrate the efficacy of this metric on an application in genealogical search.