Prototypical Cross-Attention Networks for Multiple Object Tracking and SegmentationDownload PDF

21 May 2021, 20:41 (edited 08 Nov 2021)NeurIPS 2021 SpotlightReaders: Everyone
  • Keywords: multiple object tracking and segmentation, video instance segmentation, efficient cross-attention networks, space-time memory
  • TL;DR: We design efficient cross-attention networks (PCAN) for multiple object tracking and segmentation.
  • Abstract: Multiple object tracking and segmentation requires detecting, tracking, and segmenting objects belonging to a set of given classes. Most approaches only exploit the temporal dimension to address the association problem, while relying on single frame predictions for the segmentation mask itself. We propose Prototypical Cross-Attention Network (PCAN), capable of leveraging rich spatio-temporal information for online multiple object tracking and segmentation. PCAN first distills a space-time memory into a set of prototypes and then employs cross-attention to retrieve rich information from the past frames. To segment each object, PCAN adopts a prototypical appearance module to learn a set of contrastive foreground and background prototypes, which are then propagated over time. Extensive experiments demonstrate that PCAN outperforms current video instance tracking and segmentation competition winners on both Youtube-VIS and BDD100K datasets, and shows efficacy to both one-stage and two-stage segmentation frameworks. Code and video resources are available at http://vis.xyz/pub/pcan.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: https://github.com/SysCV/pcan
13 Replies

Loading