Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognition
Abstract: Existing works in few-shot action recognition mostly fine-tune a pre-trained image model and design sophisticated temporal alignment modules at feature level. However, simply fully fine-tuning the pre-trained model could cause overfitting due to the scarcity of video samples. Additionally, we argue that the exploration of task-specific information is insufficient when relying solely on well extracted abstract features. In this work, we propose a simple but effective task-specific adaptation method (Task-Adapter) for few-shot action recognition. By introducing the proposed Task-Adapter into the last several layers of the backbone and keeping the parameters of the original pre-trained model frozen, we mitigate the overfitting problem caused by full fine-tuning and advance the task-specific mechanism into the process of feature extraction. In each Task-Adapter, we reuse the frozen self-attention layer to perform task-specific self-attention across different videos within the given task to capture both distinctive information among classes and shared information within classes, which facilitates task-specific adaptation and enhances subsequent metric measurement between the query feature and support prototypes. Experimental results consistently demonstrate the effectiveness of our proposed Task-Adapter on four standard few-shot action recognition datasets. Especially on temporal challenging SSv2 dataset, our method outperforms the state-of-the-art methods by a large margin.
Primary Subject Area: [Content] Vision and Language
Relevance To Conference: This work proposes a novel adapter-based method that leverages the prior knowledge of large-scale multimodal pre-trained models to address the challenge of few-shot action recognition, where the labeled video samples are very limited. Our work can provide insights and pave the way for new research avenues for relevant fields, prompting attention to the overlooked task-specific information and exploration of new technologies.
Submission Number: 2468
Loading