Consistency of the $k_n$-nearest neighbor rule under adaptive sampling

Robi Bhattacharjee, Geelon So, Sanjoy Dasgupta

Keywords: Smoothed analysis, online learning, non-parametric estimation, consistency

Abstract: In the adaptive sampling model of online learning, future prediction tasks can be arbitrarily dependent on the past. Every round, an adversary selects an instance to test the learner. After the learner makes a prediction, a noisy label is drawn from an underlying conditional label distribution and is revealed to both learner and adversary. A learner is consistent if it eventually performs no worse than the Bayes predictor. We study the $k_n$-nearest neighbor learner within this setting. In the worst-case, the learner will fail because an adaptive process can generate spurious patterns out of noise. However, under the mild smoothing assumption that the process generating the instances is uniformly absolutely continuous and that choice of $(k_n)_n$ is reasonable, the $k_n$-nearest neighbor rule is online consistent.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 27078