Conditional DETR for Fast Training Convergence

Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, Jingdong Wang

Abstract:

The recently-developed DETR approach applies the
transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence,
and present a conditional cross-attention mechanism for
fast DETR training. Our approach is motivated by that the
cross-attention in DETR relies highly on the content embeddings for localizing the four extremities and predicting the
box, which increases the need for high-quality content embeddings and thus the training difficulty.
Our approach, named conditional DETR, learns a conditional spatial query from the decoder embedding for
decoder multi-head cross-attention. The benefit is that
through the conditional spatial query, each cross-attention
head is able to attend to a band containing a distinct region, e.g., one object extremity or a region inside the object box. This narrows down the spatial range for localizing the distinct regions for object classification and box
regression, thus relaxing the dependence on the content embeddings and easing the training. Empirical results show
that conditional DETR converges 6.7× faster for the backbones R50 and R101 and 10× faster for stronger backbones
DC5-R50 and DC5-R101. Code is available at https:
//github.com/Atten4Vis/ConditionalDETR.