MicrobeQuest: A Multimodal Benchmark for Information Retrieval in Microbiology

ACL ARR 2025 July Submission633 Authors

Abstract: AI for Science (AI4S) is reshaping research paradigms across scientific disciplines. In microbiology, multimodal data (text, images, tables, and charts) exist in scientific literature and public databases to understand the complex relationship between diverse microbial strains and their unique traits. However, current benchmarks are either general-purpose or designed for disciplines such as material or biomedical sciences, lacking one specific for microbial sciences. Here, we developed MicrobeQuest, the first comprehensive, multimodal benchmark with 10,176 query-response pairs for microbiology-specific information retrieval to take advantage of the vast amount of available information in microbiology. We first constructed multiple rounds of manual data collection by a group of experts to curate the microbiological dataset. We then demonstrated its utility by benchmarking 19 state-of-the-art (SOTA) information retrieval (IR) methods. This yielded crucial performance insights and established a robust foundation for future IR advancements in microbiology. All benchmark resources, including code and datasets, are publicly available at https://github.com/acl-submission/MicrobeQuest.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: Resources and Evaluation

Contribution Types: Data resources

Languages Studied: English

Previous URL: https://openreview.net/forum?id=yOy9WTOtQ1

Reassignment Request Area Chair: Yes, I want a different area chair for our submission

Reassignment Request Reviewers: Yes, I want a different set of reviewers

Justification For Not Keeping Action Editor Or Reviewers: One reviewer commented they didn't find our data and no usable datasets submitted. We would like to clarify that the complete dataset is available in our GitHub page(https://github.com/acl-submission/MicrobeQuest) under the MicrobeQuest/benchmarks directory.And we are glad that reviewer acknowledges the depth of our work and confirm that we have addressed all of their concerns in rebuttal. However, they suggest that our work may be better suited for a venue allowing a more comprehensive presentation . We respectfully disagree. Our benchmark reflects this interdisciplinary vision by bridging NLP and microbiology—an area where critical knowledge remains locked in vast, unstructured literature. We believe ACL provides an ideal venue to present MicrobeQuest and its contributions to both communities. As for the AE, they appears to have overlooked our rebuttal, simply echoing Reviewer's comments. As a result, we didn't receive valuable feedback

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: No

A2 Elaboration: no risk

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: N/A

B2 Discuss The License For Artifacts: N/A

B3 Artifact Use Consistent With Intended Use: N/A

B4 Data Contains Personally Identifying Info Or Offensive Content: N/A

B5 Documentation Of Artifacts: Yes

B5 Elaboration: 3.1

B6 Statistics For Data: Yes

B6 Elaboration: 3.2

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: 4.2

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: 4.2

C3 Descriptive Statistics: Yes

C3 Elaboration: 4.4

C4 Parameters For Packages: Yes

C4 Elaboration: 4.1

D Human Subjects Including Annotators: Yes

D1 Instructions Given To Participants: N/A

D2 Recruitment And Payment: N/A

D3 Data Consent: Yes

D3 Elaboration: Appdenix C

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: Yes

D5 Elaboration: 3.2

E Ai Assistants In Research Or Writing: Yes

E1 Information About Use Of Ai Assistants: N/A

Author Submission Checklist: yes

Submission Number: 633