Universal Sequence Preconditioning

Annie Marsden, Elad Hazan

Keywords: sequence prediction, time series, pre-conditioning

TL;DR: We provide a method of preconditioning sequential data and formally prove it improves the regret bound of several methods and experimentally show it improves the performance of many models.

Abstract:

We study the problem of preconditioning in the setting of sequential prediction. From the theoretical lens of linear dynamical systems, we show that applying a convolution to the input sequence translates to applying a polynomial to the unknown transition matrix in the hidden space. With this insight, we develop a novel preconditioning method that convolves the input sequence with the coefficients of the Chebyshev or Legendre polynomials. We formally prove that this improves the regret of a wide family of prediction methods.

We proceed to apply this preconditioning technique to the method of spectral filtering. This gives the first sublinear regret bound that is also hidden-dimension free (up to logarithmic factors) even when the hidden transition matrix is asymmetric.

From rigorous experiments on synthetic data we show that our simple preconditioning method generalizes to both 1) settings where the data is \emph{not} from a linear dynamical system and 2) a broad range of learning algorithms, including recurrent neural networks.

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 22303