Efficient Matching-Free Distillation for Detection Transformers via Active Sampling

ICLR 2026 Conference Submission14316 Authors

Keywords: Knowledge+Distillation, DETR, Hungarian+Matching

Abstract: Most existing knowledge distillation approaches for DETR-based detectors depend on query matching between teacher and student models, typically utilizing Hungarian matching algorithms, which are inefficient and time consuming. To mitigate these limitations, we propose an effective and efficient distillation framework that obviates the need for matching. Specifically, we introduce a novel active sampling and alignment strategy tailored for matching-free knowledge distillation. In our approach, the output from both the teacher and student models queries are regarded as representations of their corresponding output distributions. Then, with appropriate sampling points, we concurrently sample from both distributions and then enforce consistency between the sampled outcomes, thereby aligning the distribution between teacher and student. For the sampling procedure, we devise a simple but effective attention-based sampling module, complemented by a dedicated learning strategy for effective distribution sampling. Additionally, for the selection of sampling points during distillation, we propose a prior-guided point sampling approach that more accurately captures the teacher’s output distribution, enhancing alignment with the student’s distribution. Extensive experiments conducted across multiple datasets and baseline detectors validate that our method substantially enhances the performance of the student model. Compared to the state-of-the-art DETR-Distill, our approach achieves superior performance while accelerating the distillation training process by 3.8 times. The code is available in the supplementary materials and will be publicly released upon acceptance of this paper.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 14316