OAR: Training Quantization-Friendly Object Detectors via Outlier-Aware Restriction

ICLR 2026 Conference Submission689 Authors

Keywords: Model Quantization, Object Detection, Pre-Training, Outlier Restriction, Quantization Robustness

TL;DR: Our method tackles 4/3-bit detector quantization via novel losses in pre-training. Our 4-bit model hits 35.39% mAP (↑3.03%), matching SOTA methods.

Abstract:

Model quantization is widely employed to reduce computational resource usage during inference, often in conjunction with specialized hardware system for acceleration. While modern object detectors perform well at compact bit-widths (e.g., 8-bit), further quantization to ultra-low bit-widths (e.g., 4 or 3 bits) remains challenging. We identify the presence of outliers in the statistical distribution of activations in pre-trained detectors as a key obstacle, as such outliers expand the dynamic range and increase quantization error. Moreover, we observe significant numerical discrepancies in activation outliers across the task branches of the detection head, potentially leading to imbalanced sub-task performance after quantization.
To address these issues, we propose Resonant Shrinkage loss and Output Adaptation to suppress activation outliers during pre-training. Additionally, we introduce Edge-Aware KURE loss to enhance the robustness of detector weights under quantization. All components are applied during the pre-training phase, producing detectors that are more quantization-friendly without altering training hyperparameters, while also reducing quantization sensitivity disparities between task branches. Our training framework is compatible with existing state-of-the-art quantization methods and delivers improved performance. Notably, even with naive post-training quantization requiring only a small calibration set, our 4-bit quantized ATSS model achieves 35.39\% mAP, outperforming the original quantized version by 3.03\%. Code will be released soon.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 689