An Integrated Multi-Label Multi-Modal Framework in Deep Metric Learning

Natalie Dullerud, Karsten Roth, Haoran Zhang, Qixuan Jin, Thomas Hartvigsen, Marzyeh Ghassemi

Keywords: deep metric learning, healthcare, multimodal, multilabel

Abstract: Domains such as healthcare demand machine learning models which provide representations for complex relationships between both heterogeneous modes of data, and multiple co-occurring labels. Previous works have tackled representation learning in the multi-label, multi-modal setting, but have neglected to consider the common requirement of generalization to novel, and unknown, tasks at test-time. In this work, we propose an integrated multi-modal multi-label framework for deep metric learning, which we term 3ML--DML. Our framework extends existing proxy learning losses to the multi-label domain, and provides a novel method for enforcement of label correlations via these proxies. The multi-modal component builds a standard fusion model but draws from deep metric learning criteria in order to incorporate auxiliary, high-dimensional embedding and feature spaces from each mode of data as context to match with the output of the fusion model. We explore our method in a variety of settings, including on healthcare data, and demonstrate improvement over constructed baselines both in the context of multi-label multi-modal learning but most poignantly, in zero-shot generalization to new labels.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning