Multi-Modal Data Exploration via Language Agents

ACL ARR 2025 July Submission153 Authors

Abstract:

International enterprises, organizations, and hospitals collect large amounts of multi-modal data stored in databases, text documents, images, and videos. While there has been recent progress in the separate fields of multi-modal data exploration as well as in database systems that automatically translate natural language questions to database query languages, the research challenge of querying both structured databases and unstructured modalities (e.g., texts, images) in natural language remains largely unexplored.
In this paper, we propose M$^2$EX -- a system that enables multi-modal data exploration via language agents. Our approach is based on the following research contributions: (1) Our system is inspired by a real-world use case that enables users to explore multi-modal information systems. (2) M$^2$EX leverages an LLM-based agentic AI framework to decompose a natural language question into subtasks such as text-to-SQL generation and image analysis and to orchestrate modality-specific experts in an efficient query plan. (3) Experimental results on multi-modal datasets, encompassing relational data, text, and images, demonstrate that our system outperforms state-of-the-art multi-modal exploration systems, excelling in both accuracy and various performance metrics, including query latency, API costs, and planning efficiency, thanks to the more effective utilization of the reasoning capabilities of LLMs.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: Multi-Modality, Data exploration, Database, LLM, Language Agents

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources

Languages Studied: English

Previous URL: https://openreview.net/forum?id=LoKkvUQwnu

Reassignment Request Area Chair: Yes, I want a different area chair for our submission

Reassignment Request Reviewers: Yes, I want a different set of reviewers

Justification For Not Keeping Action Editor Or Reviewers: Dismissing the work without any concrete comments regarding correctness of the results or argumentation, limited perceived impact of the methods or findings, lack of clarity .

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: N/A

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: Section 3, 4 and 4.1

B2 Discuss The License For Artifacts: N/A

B2 Elaboration: N/A

B3 Artifact Use Consistent With Intended Use: N/A

B4 Data Contains Personally Identifying Info Or Offensive Content: N/A

B5 Documentation Of Artifacts: Yes

B5 Elaboration: Section 4

B6 Statistics For Data: Yes

B6 Elaboration: Section 4.1

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: Section 4.2

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: Section 4.2

C3 Descriptive Statistics: N/A

C4 Parameters For Packages: Yes

C4 Elaboration: Section 4.2, and 4.3

D Human Subjects Including Annotators: No

D1 Instructions Given To Participants: N/A

D2 Recruitment And Payment: N/A

D3 Data Consent: N/A

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: N/A

E Ai Assistants In Research Or Writing: No

E1 Information About Use Of Ai Assistants: N/A

Author Submission Checklist: yes

Submission Number: 153