Limited access to previous data is challenging when using Gaussian process (GP) models for sequential learning. This results in inaccuracies in posterior, hyperparameter learning, and inducing variables. The recently proposed ‘dual’ sparse GP model enables inference of variational parameters in such a setup. In this paper, using the dual GP, we tackle the problem arising due to a lack of access to previous data for estimating hyperparameters of a sparse Gaussian process. We propose utilizing the concept of ‘memory’. To pick representative memory, we develop the ‘Bayesian leverage score’ built on the ridge leverage score. We experiment and perform an ablation study with a sequential learning data set, split MNIST, to showcase the usefulness of the proposed method.

Sequential Learning in GPs with Memory and Bayesian Leverage Score

With the latest advances in deep learning, several methods have been investigated for optimal learning settings in scenarios where the data stream is continuous over time. However, training sparse networks in such settings has often been overlooked. In this paper, we explore the problem of training a neural network with a target sparsity in a particular case of online learning: the anytime learning at macroscale paradigm (ALMA). We propose a novel way of progressive pruning, referred to as \textit{Anytime Progressive Pruning} (APP); the proposed approach significantly outperforms the baseline dense and Anytime OSP models across multiple architectures and datasets under short, moderate, and long-sequence training. Our method, for example, shows an improvement in accuracy of $\approx 7\%$ and a reduction in the generalization gap by $\approx 22\%$, while being $\approx 1/3$ rd the size of the dense baseline model in few-shot restricted imagenet training. 

APP: Anytime Progressive Pruning

We present two approaches to approximate online Bayesian inference for the parameters of DNNs. Both are based on diagonal Gaussian approximations and linearize the network at each step to ensure efficient computation.
The first approach optimizes the exclusive KL,  KL(p,q); this amounts to matching the marginal mean and {\em precision} of p and q.
The second approach  optimizes the inclusive KL,  KL(q,p), which amounts to matching the marginal mean and {\em variance} of p and q. The latter approach turns out to be equivalent to the previously proposed ``fully decoupled EKF'' approach.
We show experimentally that exclusive KL is more effective than both inclusive KL and one-pass SGD.

On diagonal approximations to the extended Kalman filter for online training of Bayesian neural networks

Temporal point process serves as an essential tool for modeling time-to-event data in continuous time space. Despite having massive amounts of event sequence data from various domains like social media, healthcare etc., real world application of temporal point process are not capable of thriving in continually evolving environment with minimal supervision while retaining previously learnt knowledge. To tackle this, we propose HyperHawkes, a hypernetwork based continually learning temporal point process for continuous modeling of time-to-event sequences with minimal forgetting. We demonstrate the application of the proposed framework through our experiments on two real-world datasets.

Continual Learning for Time-to-Event Modeling

A healthy human brain manifests a high variance in signals captured from different modalities like EEG, MEG, and fMRI during ongoing activity. Further, there is brain signal variability exhibited at the user level. The study of within-subject and cross-subject variance is essential to design general Brain-Computer Interfaces (BCI) that help interpret these signals into intended outcomes. We propose that these variations can be studied under the umbrella of Continual Learning (CL). We performed an empirical evaluation to understand the impact of CL strategies on the benchmark dataset. Our findings in within-subject and cross-subject scenarios suggest that CL strategies can outperform offline learning and build robust models for BCI applications.
In the cross-subject scenario, CL can lead to learning invariant subject representation when transferring knowledge from one subject to another. In the within-subject scenario, CL can enhance performance when transferring knowledge from one session to another.

Continual Learning for EEG based Brain Computer Interfaces

Continual Learning is important for real-world natural language processing applications,
where computational systems are required to interact with continuous streams of tasks
and language over time. When forced to adapt to new tasks and inputs, language models
experience catastrophic forgetting. The current generative replay-based algorithms are not
scalable to many tasks, and their performance may degrade from a change in the task order.
In this paper, we propose a model based on network growth - a pre-trained Transformer
with Adapter modules for each task - that sequentially learns new NLP tasks in various
domains and prevents catastrophic forgetting without retraining the model from scratch.
We train and maintain light weight adapter modules sequentially for each task. Without
increasing network growth by more than 15% and avoiding replay and task order bias,
the current design allows us to increase average task accuracy by 1.3% over the baseline
models.

ACML 2022 Workshop CLL Submissions

Sequential Learning in GPs with Memory and Bayesian Leverage Score

APP: Anytime Progressive Pruning

On diagonal approximations to the extended Kalman filter for online training of Bayesian neural networks

Continual Learning for Time-to-Event Modeling

Continual Learning for EEG based Brain Computer Interfaces

Lifelong Language Learning with Adapter based Transformers