Multi-Granularity Hand Action Detection
Abstract: Detecting hand actions in videos is crucial for understanding video content and has diverse real-world applications. Existing approaches often focus on whole-body actions or coarse-grained action categories, lacking fine-grained hand-action localization information. To fill this gap, we introduce the FHA-Kitchens (Fine-Grained Hand Actions in Kitchen Scenes) dataset, providing both coarse- and fine-grained hand action categories along with localization annotations. This dataset comprises 2,377 video clips and 30,047 frames, annotated with approximately 200k bounding boxes and 880 action categories. Evaluation of existing action detection methods on FHA-Kitchens reveals varying generalization capabilities across different granularities. To handle multi-granularity in hand actions, we propose MG-HAD, an End-to-End Multi-Granularity Hand Action Detection method. It incorporates two new designs: Multi-dimensional Action Queries and Coarse-Fine Contrastive Denoising. Extensive experiments demonstrate MG-HAD's effectiveness for multi-granularity hand action detection, highlighting the significance of FHA-Kitchens for future research and real-world applications. The dataset and source code will be released.
Primary Subject Area: [Content] Vision and Language
Secondary Subject Area: [Content] Vision and Language, [Experience] Multimedia Applications
Relevance To Conference: Our work delves into hand action understanding based on video and image data, making improvements in both action localization boxes and category descriptions. We introduce a finer-grained hand action dataset (FHA-Kitchens) for the multimedia domain and further propose a new baseline method (MG-HAD) suitable for multi-granularity hand actions. We advance the fusion of vision and language in different ways and present a new solution for multimedia content understanding, particularly focusing on hand-action research.
Supplementary Material: zip
Submission Number: 1656
Loading