From Coarse to Fine-grained Concept based Discrimination for Phrase Detection

Maan Qraitem, Bryan A. Plummer

Keywords: Phrase Detection, Vision Language Understanding

Abstract: Phrase detection requires methods to identify if a phrase is relevant to an image and localize it if applicable. A key challenge in training more discriminative phrase detection models is sampling negatives. However, sampling techniques from prior work focus primarily on hard, often noisy, negatives disregarding the broader distribution of negative samples. To address this problem, we introduce CFCD-Net, a phrase detector that differentiates between phrases through two novels methods. First, we generate groups that consist of semantically similar words we call concepts (\eg \{dog, cat, horse, ...\} vs.\ \{car, truck, ...\}), and then train our CFCD-Net to discriminate between a region of interest and its unrelated concepts. Second, for phrases containing fine-grained mutually-exclusive words (\eg colors), we force the model into selecting only one applicable phrase for each region using our novel fine grained module (FGM). We evaluate our approach on the Flickr30K Entities and RefCOCO+ datasets, where we improve mAP over the state-of-the-art by 1.5-2 points. When considering only the phrases affected by our fine-grained reasoning module, we improve by 3-4 points on both datasets

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Applications (eg, speech processing, computer vision, NLP)