Fisher-aware Quantization for DETR Detectors with Critical-category Objectives

Huanrui Yang, Yafeng Huang, Zhen Dong, Yizhao Zhang, Denis A Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Yuan Du, Kurt Keutzer, Shanghang Zhang

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Quantization, Detection Transformers, Fisher information, Finegrained performance

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: This work improves critical category performance in quantized DETR with Fisher-aware quantization scheme and Fisher-trace regularization

Abstract: The impact of quantization on the overall performance of deep learning models is a well-studied problem. However, understanding and overcoming its effects on a more fine-grained level is still lacking, especially for harder tasks such as object detection with both classification and regression objectives. This work identifies the performance for a subset of task-critical categories, i.e. the critical-category performance, as a crucial yet largely overlooked fine-grained objective for detection tasks. We analyze the impact of quantization at the category-level granularity, and propose methods to improve performance for the critical categories. Specifically, we find that certain critical categories have a higher sensitivity to quantization, and have inferior generalization after quantization-aware training (QAT). To explain this, we provide theoretical and empirical links between their performance gaps and the corresponding loss landscapes with the Fisher information framework. Using this evidence, we propose a Fisher-aware mixed-precision quantization scheme, and a Fisher-trace regularization for the QAT on the critical-category loss landscape. The proposed methods improve critical-category performance metrics of the quantized transformer-based DETR detectors. When compared to the conventional quantization objective, our Fisher-aware quantization scheme shows up to 0.9% mAP increase on COCO dataset. A further 0.5% mAP improvement is achieved for selected critical categories with the proposed Fisher-trace regularization.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4046