Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrieval
Abstract: To train image-caption retrieval (ICR) methods, contrastive loss functions are a common choice for optimization functions.
Unfortunately, contrastive ICR methods are vulnerable to predictive feature suppression.
Predictive features are features that correctly indicate the similarity between a query and a candidate item.
However, in the presence of multiple predictive features during training, encoder models tend to suppress redundant predictive features, since these features are not needed to learn to discriminate between positive and
negative pairs.
We introduce an approach to reduce predictive feature suppression for resource-constrained ICR methods: latent target decoding (LTD). We add an additional decoder to the contrastive ICR framework, to reconstruct the input caption in a latent space of a general-purpose
sentence encoder, which prevents the image and caption encoder from suppressing
predictive features.
We implement the LTD objective as an optimization constraint, to ensure that the reconstruction loss is below a bound value while primarily optimizing for the contrastive loss.
Importantly, LTD does not depend on additional training data or expensive (hard) negative mining strategies.
Our experiments show that, unlike reconstructing the input caption in the input space, LTD reduces predictive feature suppression, measured by obtaining higher recall@k, r-precision, and nDCG scores than a contrastive ICR baseline.
Moreover, we show that LTD should be implemented as an optimization constraint instead
of a dual optimization objective. Finally, we show that LTD can be used with different
contrastive learning losses and a wide variety of resource-constrained ICR methods.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: We would like to thank all the reviewers for their reviews and constructive feedback. We have uploaded the camera ready of the paper. The main points addressed in the paper (compared to the previous version):
- Fixed some typos and notation mistakes.
- Changed the references.
Video: https://www.youtube.com/watch?v=CrT7wlzY5K8
Code: https://github.com/MauritsBleeker/reducing-predictive-feature-suppression
Assigned Action Editor: ~Yonatan_Bisk1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 725
Loading