Taylorformer: Probabalistic Modelling for Random Processes including Time Series
Keywords: stochastic processes, probabilistic machine learning, uncertainty, attention
TL;DR: An attention-based model approximating a stochastic process which incorporates insights from Gaussian Processes and Taylor series to better random processes including time-series.
Abstract: We propose the Taylorformer for random processes such as time series. Its two key components are: 1) the LocalTaylor wrapper which adapts Taylor approximations (used in dynamical systems) for use in neural network-based probabilistic models, and 2) the MHA-X attention block which makes predictions in a way inspired by how Gaussian Processes' mean predictions are linear smoothings of contextual data. Taylorformer outperforms the state-of-the-art in terms of log-likelihood on 5/6 classic Neural Process tasks such as meta-learning 1D functions, and has at least a 14\% MSE improvement on forecasting tasks, including electricity, oil temperatures and exchange rates. Taylorformer approximates a consistent stochastic process and provides uncertainty-aware predictions. Our code is provided in the supplementary material.
Submission Number: 45
Loading