Hierarchical Multimodal Knowledge Matching for Training-Free Open-Vocabulary Object Detection
Keywords: open-vocabulary object detection, multimodal knowledge, vision and language
Abstract: Open-Vocabulary Object Detection (OVOD) aims to leverage the generalization capabilities of pre-trained vision language models for detecting objects beyond the trained categories.
Existing methods mostly focus on supervised learning strategies based on available training data, which might be suboptimal for data-limited novel categories.
To tackle this challenge, this paper presents a $\textbf{H}$ierarchical $\textbf{M}$ultimodal $\textbf{K}$nowledge $\textbf{M}$atching method ($\textbf{HMKM}$) to better represent novel categories and match them with region features.
Specifically, HMKM includes a set of object prototype knowledge that is obtained using limited category-specific images, acting as off-the-shelf category representations.
In addition, HMKM also includes a set of attribute prototype knowledge to represent key attributes of categories at a fine-grained level, with the goal to distinguish one category from its visually similar ones.
During inference, two sets of object and attribute prototype knowledge are adaptively combined to match categories with region features.
The proposed HMKM is training-free and can be easily integrated as a plug-and-play module into existing OVOD models.
Extensive experiments demonstrate that our HMKM significantly improves the performance when detecting novel categories across various backbones and datasets.
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9923
Loading