No Need for Ad-hoc Substitutes: The Expected Cost is a Principled All-purpose Classification Metric
Abstract: The expected cost (EC) is one of the main classification metrics introduced in statistical and machine learning books. It is based on the assumption that, for a given application of interest, each decision made by the system has a corresponding cost which depends on the true class of the sample. An evaluation metric can then be defined by taking the expectation of the cost over the data. Two special cases of the EC are widely used in the machine learning literature: the error rate (one minus the accuracy) and the balanced error rate (one minus the balanced accuracy or unweighted average recall). Other instances of the EC can be useful for applications in which some types of errors are more severe than others, or when the prior probabilities of the classes differ between the evaluation data and the use-case scenario. Surprisingly, the general form for the EC is rarely used in the machine learning literature. Instead, alternative ad-hoc metrics like the F-beta score and the Matthews correlation coefficient (MCC) are used for many applications. In this work, we argue that the EC is superior to these alternative metrics, being more general, interpretable, and adaptable to any application scenario. We provide both theoretically-motivated discussions as well as examples to illustrate the behavior of the different metrics.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: This revision addresses the comments from the reviewers. The changes are highlighted in red in the pdf and are explained in detail in the responses to each of the reviewers. Here we summarize the main changes:
* We moved the comparison with various standard classification metrics to the main body of the text
* We added a new section with an analysis of results on several real datasets
* We added a comment on the use of the EC as objective function for optimization
* We added a discussion on how the standard metrics can be interpreted as having effective costs for each type of error that depend on the system under evaluation
Assigned Action Editor: ~Daniel_M_Roy1
Submission Number: 1917
Loading