Weight Factorization Based Incremental Learning in Generalized Few Shot Segmentation

Anuska Roy, Viswanath Gopalakrishnan

Abstract: Generalized Few-shot Semantic Segmentation (GFSS) targets to segment novel object categories using a few annotated examples after learning the segmentation on a set of base classes. A typical GFSS training involves two stages - base class learning followed by novel class addition and learning. While existing methods have shown promise, they often struggle when novel classes are significant in number. Most current approaches freeze the encoder backbone to retain base class accuracy; however, freezing the encoder backbone can potentially impede the assimilation of novel information from the new classes. To address this challenge, we propose to use an incremental learning strategy in GFSS for learning both encoder backbone and novel class prototypes. Inspired by the recent success of Low Rank Adaptation techniques (LoRA), we introduce incremental learning to the GFSS encoder backbone with a novel weight factorization method. Our newly proposed rank adaptive weight merging strategy is se

External IDs:dblp:conf/visigrapp/RoyG25