PAC-Bayesian Bounds for Learning LTI-ss systems with Input from Empirical Loss

Deividas Eringis, john leth, Rafal Wisniewski, Zheng-Hua Tan, Mihaly Petreczky

Keywords: LTI, PAC, PAC-Bayesian, generalisation gap, bounds

TL;DR: PAC-Bayesian bounds on generalisation gap for LTI systems, assuming data is generated by an LTI system.

Abstract:

In this paper we derive a Probably Approximately Correct(PAC)-Bayesian error bound for linear time-invariant (LTI) stochastic dynamical systems with inputs. Such bounds
are widespread in machine learning, and they are useful for characterizing the predictive power of models learned from
finitely many data points.
In particular, the bound derived in this paper relates
future average prediction errors with the prediction error
generated by the model on the data used for learning.
In turn, this allows us to provide finite-sample error bounds for
a wide class of learning/system identification algorithms.
Furthermore, as LTI systems are a sub-class of recurrent neural
networks (RNNs), these error bounds could be a first step towards
PAC-Bayesian bounds for RNNs.

Submission Number: 72