Multil-Level Multimodal Alignment with Knowledge-Guided Instance-Wise Discrimination
Keywords: Multimodal learning, Meta Learning, Graph representation learning, Contrastive learning, Knowledge-Guided Instance-Wise Discrimination
Abstract: In multimodal alignment, meta-alignment and multi-level alignment play important roles. However, it is challenging to integrate meta-alignment into a multi-level multimodal alignment framework involving the operation on both reducible substances (e.g., molecules and spectrum) and irreducible elements (e.g., atoms and spectral peaks). It not only inherits the challenges from meta-alignment (e.g., heterogeneity, loss of nuance, interference, and conflicting similarities) but also introduces new challenges: navigating the interactions among reducible substances and irreducible elements and recognizing objects at each level. Many existing alignment methods suffer from inaccurate component relation estimation and potential bias, as they hold manual definitions of pair closeness. In response, we introduce Multi-level Multimodal Alignment with Knowledge-guided Instance-wise Discrimination (K-M3AID), an innovative approach that utilizes continuous knowledge variables with inherent natural ordering for meta-alignment. K-M3AID effectively addresses these challenges by promoting both reliable distance learning and unbiased alignment within the context of cross-modality alignment for multi-level structures. Extensive empirical studies conducted on complex molecular structures underscore the substantial efficacy of K-M3AID. It significantly improves matching accuracy while augmenting multi-level alignment capabilities. This novel approach holds great promise for advancing alignment techniques across diverse molecular contexts, offering a more robust foundation for ongoing research in chemical analysis and beyond.
Supplementary Material: zip
Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4391
Loading