Geometric Implications of Classification on Reducing Open Space Risk

Matthew Lau, Leyan Pan, Stefan Davidov, Athanasios P Meliopoulos, Wenke Lee

Keywords: Machine Learning, Classification, Open Space Risk, Anomaly Detection

TL;DR: Through a geometric lens, we show that linear classifiers can have arbitrarily high VC dimension but can be designed to control open space risk to reject anomalous inputs effectively.

Abstract:

To reduce open space risk of hypotheses, we reexamine the 'simplest' hypothesis class, binary linear classifiers, geometrically.
Generalizing linear classification,
we establish a surprising fact: linear classifiers can have arbitrarily high VC dimension, stemming from increasing the number of partitions in input space.
Hence, linear classifiers with multiple margins are more expressive than single-margin classifiers.
Despite a higher VC dimension, such classifiers have less open space risk than halfspace separators.
These geometric insights are useful to detect unseen classes, while probabilistic modeling of risk minimization helps with seen classes.
In supervised anomaly detection, we show that a classifier that combines a probabilistic and geometric lens can detect both seen and unseen anomalies well.

Submission Number: 171