HEAV: Hierarchical Ensembling of Augmented Views for Image Captioning

Chia-Wen Kuo, Zsolt Kira

Keywords: image captioning, vision and language

Abstract: A great deal of progress has been made in image captioning, driven by research into how to encode the image using pre-trained models. This includes visual encodings (e.g. image grid features or detected objects) and more recently textual encodings (e.g. image tags or text descriptions of image regions). As more advanced encodings are available and incorporated, it is natural to ask: how to efficiently and effectively leverage and ensemble the heterogeneous set of encodings? In this paper, we propose to regard the encodings as augmented views of the input image. The model encodes each view independently with a shared encoder efficiently, and a contrastive loss is incorporated across the encoded views to improve the representation quality, as well as to enable semi-supervised training of image captioning. Our proposed hierarchical decoder then adaptively ensembles the encoded views according to their usefulness by first ensembling within each view at the token level, and then across views at the view level. We demonstrate significant performance improvements of +5.6% CIDEr on MS-COCO compared to state of the art under the same trained-from-scratch setting and +16.8% CIDEr on Flickr30K with semi-supervised training, and conduct rigorous analyses to demonstrate the importance of each part of our design.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Applications (eg, speech processing, computer vision, NLP)

TL;DR: We tackle the problem of how to efficiently and effectively leverage and ensemble heterogeneous views for image captioning