Birds of an Odd Feather: Guaranteed Out-of-Distribution (OOD) Novel Category Detection

Yoav Wald, Suchi Saria

Keywords: ood detection, novel category, distribution shift, novel class, novel subgroup, constrained learning

TL;DR: Solution for the problem of novel category detection under distribution shift, based on constrained learning; Guaranteed solution under relatively mild assumptions

Abstract:

In this work, we solve the problem of novel category detection under distribution shift. This problem is critical to ensuring the safety and efficacy of machine learning models, particularly in domains such as healthcare where timely detection of novel subgroups of patients is crucial.

To address this problem, we propose a method based on constrained learning. Our approach is guaranteed to detect a novel category under a relatively weak assumption, namely that rare events in past data have bounded frequency under the shifted distribution. Prior works on the problem do not provide such guarantees, as they either attend to very specific types of distribution shift or make stringent assumptions that limit their guarantees.

We demonstrate favorable performance of our method on challenging novel category detection problems over real world datasets.