Using Local Complexity to Evaluate Out-of-Distribution Generalization

Grace O'Brien, Andrew Aguilar, Robert Jasper, Henry Kvinge, Sarah McGuire Scullen, Helen Jenne

Track: Extended Abstract (non-archival, 4 pages)

Keywords: out-of-distribution generalization, local complexity, robustness, deep learning generalization, data-dependent complexity measures

TL;DR: We investigate local complexity as a predictor of model performance on out-of-distribution data.

Abstract: Despite their growing ubiquity, the inner workings of deep neural networks are still largely a black box. Even in the case of classification tasks, common methods used to assess model performance do not give insight into whether the model will generalize to unseen data. In this extended abstract, we investigate local complexity (LC) (Humayun et al., 2024b), a geometric measure of the input space, as a predictor of model performance on out-of-distribution (OOD) data. We find that LC alone is not sufficient to predict model generalization, but that it does capture meaningful information about the correctness of individual predictions, suggesting it may be useful as part of a larger set of tools to understand OOD generalization.

Submission Number: 24