AKORN: Adaptive Knots generated Online for RegressioN splines

Sunil Madhow, Dheeraj Baby, Yu-Xiang Wang

TL;DR: We propose a knot-selection scheme for producing a knot set whose corresponding least-squares regression spline is optimal over the space of functions in TV_1

Abstract: In order to attain optimal rates, state-of-the-art algorithms for non-parametric regression require that a hyperparameter be tuned according to the smoothness of the ground truth (Tibshirani, 2014). This amounts to an assumption of oracle access to certain features of the data-generating process. We present a parameter-free algorithm for offline non-parametric regression over $TV_1$-bounded functions. By feeding offline data into an optimal online denoising algorithm styled after (Baby et al., 2021), we are able to use change-points to adaptively select knots that respect the geometry of the underlying ground truth. We call this procedure AKORN (Adaptive Knots gener- ated Online for RegressioN splines). By combining forward and backward passes over the data, we obtain an estimator whose empirical performance is close to Trend Filtering (Kim et al., 2009; Tibshirani, 2014), even when we provide the latter with oracle knowledge of the ground truth’s smoothness.

Lay Summary: This paper introduces AKORN, a method for drawing smooth curves through noisy data—like temperature trends or stock prices—without needing to guess how complex the pattern is. Unlike traditional tools that require manual tuning, AKORN learns where to adjust its fit by first selectively hiding parts of the data from itself. It places curve “bends” only where needed, adapting automatically to the data’s structure. Despite being fully automatic, it performs nearly as well as methods that rely on expert tuning.

Link To Code: https://github.com/SunilMadhow/AKORN

Primary Area: Theory->Everything Else

Keywords: nonparametric regression, total variation, adaptive, splines, online learning

Submission Number: 7962