From IID to the Independent Mechanisms assumption in continual learning

Oleksiy Ostapenko, Alexandre Lacoste, Laurent Charlin

Keywords: Continual learning, independent mechanisms, modularity, compositionality

TL;DR: We propose to replace the i.i.d. assumption with independent mechanisms assumption.

Abstract: Current machine learning algorithms are successful in learning clearly defined tasks from large i.i.d. data. Continual learning (CL) requires learning without iid-ness and developing algorithms capable of knowledge retention and transfer, the latter can be boosted through systematic generalization. Dropping the i.i.d. assumption requires replacing it with another hypothesis. While there are several candidates, here we advocate that the independent mechanism assumption (IM) (Scho ̈lkopf et al. 2012) is a useful hypothesis for representing knowledge in a form, that makes it easy to adapt to new tasks in CL. Specifically, we review several types of distribution shifts that are common in CL and point out in which way a system that represents knowledge in form of causal modules may outperform monolithic counterparts in CL. Intuitively, the efficacy of IM solution emerges since: (i) causal modules learn mechanisms invariant across domains; (ii) if causal mechanisms must be up- dated, modularity can enable efficient and sparse updates.