From IID to the Independent Mechanisms assumption in continual learningDownload PDF

Published: 11 Jan 2023, Last Modified: 05 May 2023AAAI23 Bridge Continual CausalityReaders: Everyone
Keywords: Continual learning, independent mechanisms, modularity, compositionality
TL;DR: We propose to replace the i.i.d. assumption with independent mechanisms assumption.
Abstract: Current machine learning algorithms are successful in learning clearly defined tasks from large i.i.d. data. Continual learning (CL) requires learning without iid-ness and developing algorithms capable of knowledge retention and transfer, the latter can be boosted through systematic generalization. Dropping the i.i.d. assumption requires replacing it with another hypothesis. While there are several candidates, here we advocate that the independent mechanism assumption (IM) (Scho ̈lkopf et al. 2012) is a useful hypothesis for representing knowledge in a form, that makes it easy to adapt to new tasks in CL. Specifically, we review several types of distribution shifts that are common in CL and point out in which way a system that represents knowledge in form of causal modules may outperform monolithic counterparts in CL. Intuitively, the efficacy of IM solution emerges since: (i) causal modules learn mechanisms invariant across domains; (ii) if causal mechanisms must be up- dated, modularity can enable efficient and sparse updates.
5 Replies