OmniEarth-Bench: Probing Cognitive Abilities of MLLMs for Earth's Multi-sphere Observation Data

Fengxiang Wang, Mingshuo Chen, Xuming He, YiFan Zhang, Yueying Li, Feng Liu, Zijie Guo, Zhenghao Hu, Jiong WANG, Jingyi Xu, Zhangrui Li, Junchao Gong, Di Wang, Fenghua Ling, Ben Fei, Weijia Li, Long Lan, Wenjing Yang, Jing Zhang, Wenlong Zhang et al. (1 additional authors not shown)

Keywords: Earth spheres, Observation Earth Data, MLLM, Benchmark

Abstract: Existing benchmarks for multimodal learning in Earth science offer limited, siloed coverage of Earth’s spheres and their cross-sphere interactions, typically restricting evaluation to the human-activities sphere or atmosphere and to at most 16 tasks. Holistically evaluating MLLMs on observational data across all Earth spheres face three limitation: multi-source heterogeneous data, unlocking scientific formulation and cross-sphere reasoning. Therefore, we introduce OmniEarth-Bench, the first multimodal benchmark that systematically spans all six spheres—atmosphere, lithosphere, oceansphere, cryosphere, biosphere, and human-activity sphere—and cross-sphere. Built with a scalable, modular pipeline that ingests 33 native Earth-observation sources and expert-in-the-loop curation, OmniEarth-Bench produces 29,855 standardized, expert-curated annotations. All annotations are organized into a four-level hierarchy (Sphere, Scenario, Ability, Task), encompassing 109 expert-curated evaluation tasks. Experiments on 9 state-of-the-art MLLMs reveal that even the most advanced models struggle with our benchmarks, where none of them reach 35% accuracy, revealing systematic gaps in Earth-system cognitive ability. The dataset and evaluation code were released.

Primary Area: datasets and benchmarks

Submission Number: 8406