T2S: Tokenized Skill Scaling for Lifelong Imitation Learning

Hongquan Zhang

Abstract:

Abstract—The main challenge in lifelong imitation learning
lies in the balance between mitigating catastrophic forgetting of
previous skills while maintaining sufficient capacity for acquiring
new ones. However, current approaches typically address these
aspects in isolation, overlooking their internal correlation in
lifelong skill acquisition. We address this limitation with a unified
framework named Tokenized Skill Scaling (T2S). Specifically, by
tokenizing the model parameters, the linear parameter mapping
of the traditional transformer is transformed into cross-attention
between input and learnable tokens, thereby enhancing model
scalability through the easy extension of new tokens. Additionally,
we introduce language-guided skill scaling to transfer knowledge
across tasks efficiently and avoid linearly growing parameters.
Extensive experiments across diverse tasks demonstrate that
T2S: 1) effectively prevents catastrophic forgetting (achieving an
average NBT of 1.0% across the three LIBERO task suites), 2)
excels in new skill scaling with minimal increases in trainable
parameters (needing only 8.0% trainable tokens in an average
of lifelong tasks), and 3) enables efficient knowledge transfer
between tasks (achieving an average FWT of 77.7% across the
three LIBERO task suites), offering a promising solution for
lifelong imitation learning.