MV-CLAM: Multi-View Molecular Interpretation with Cross-Modal Projection via Language Model
Keywords: Molecule captioning, large language models, drug discovery, molecule representation learning
TL;DR: MV-CLAM introduces MQ-Former, a model that aligns 2D and 3D molecular representations with text via a novel cross-modal projector, improving tasks like molecule-text retrieval, captioning, and question answering.
Abstract: Large language models (LLMs) have shown significant potential in the biomolecular domain, particularly by demonstrating that effective adaptation of molecular representations for LLMs can greatly improve the quality of molecular captions.
Most previous works have focused on aligning unimodal molecular structures with text, overlooking the diversity of modalities.
Naive approaches to aligning multi-modal molecular structures with text often lead to (1) separately aligned embeddings, (2) inconsistent textual representations, and (3) increased computational overhead.
To address these challenges, we propose LLM framework MV-CLAM equipped with MQ-Former, a novel multi-querying transformer.
This architecture introduces a cross-model projector facilitating the simultaneous alignment of 2D and 3D molecular representations to a unified text token.
By employing a shared self-attention layer, MQ-Former preserves rich molecular embeddings across different dimensions while consolidating them into a universal molecular token.
Our approach outperforms baseline models in both molecule-text retrieval and molecule captioning tasks.
Additionally, our framework shows promising results for zero-shot molecule editing, showcasing its capacity to extend beyond description generation.
By effectively integrating multi-view molecular data into a format conducive to LLMs, our method serves as a valuable tool for enhancing the characterization and understanding of chemical structures, facilitating a more seamless transition from molecular data to textual descriptions. The source code of MV-CLAM is available in https://github.com/sumin124/mv-clam.git.
Submission Number: 134
Loading