Using Platt’s scaling for calibration after undersampling — limitations and how to address them

TMLR Paper5136 Authors

Abstract: When modelling data where the response is dichotomous and highly imbalanced, response-based sampling where a subset of the majority class is retained (i.e., undersampling) is often used to create more balanced training datasets prior to modelling. However, the models fit to this undersampled data, which we refer to as base models, generate predictions that are severely biased. There are several calibration methods that can be used to combat this bias, one of which is Platt’s scaling. Here, a logistic regression model is used to model the relationship between the base model’s original predictions and the response. Despite its popularity for calibrating models after undersampling, Platt’s scaling was not designed for this purpose. Our work presents what we believe is the first detailed study focused on the validity of using Platt’s scaling to calibrate models after undersampling. We show analytically, as well as via a simulation study, that Platt’s scaling should not be used for calibration after undersampling without critical thought. If Platt’s scaling would have been able to successfully calibrate the base model had it been trained on the entire dataset (i.e., without undersampling), then Platt’s scaling might be appropriate for calibration after undersampling. If this is not the case, we recommend a modified version of Platt’s scaling that fits a logistic generalized additive model to the logit of the base model’s predictions, as this method is theoretically motivated and performed relatively well across the settings considered in our study.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission: We have made the changes that we indicated we would make in our responses to the reviewers. More substantial changes include 1) adding more simulation runs to allow for consideration of variability in the results; 2) fitting a random forest model to the wildland fire data; and 3) restructuring of Section 2 to emphasize Platt's scaling, as opposed to calibration methods more generally.

Assigned Action Editor: ~Geoff_Pleiss1

Submission Number: 5136