NeurIPS 2024 Datasets and Benchmarks Track Submissions

The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning
Ruben Ohana, Michael McCabe, Lucas Thibaut Meyer, Rudy Morel, Fruzsina Julia Agocs, Miguel Beneitez, Marsha Berger, Blakesley Burkhart, Stuart B. Dalziel, Drummond Buschman Fielding, Daniel Fortunato, Jared A. Goldberg, Keiya Hirashima, Yan-Fei Jiang, Rich Kerswell, Suryanarayana Maddu, Jonah M. Miller, Payel Mukhopadhyay, Stefan S. Nixon, Jeff Shen et al. (6 additional authors not shown)
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
Do Counterfactually Fair Image Classifiers Satisfy Group Fairness? -- A Theoretical and Empirical Study
Sangwon Jung, Sumin Yu, Sanghyuk Chun, Taesup Moon
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers
Hussein Mozannar, Valerie Chen, Mohammed Alsobay, Subhro Das, Sebastian Zhao, Dennis Wei, Manish Nagireddy, Prasanna Sattigeri, Ameet Talwalkar, David Sontag
- 28 May 2024 (modified: 13 Nov 2024)
- Submitted to NeurIPS 2024 Track Datasets and Benchmarks
- Readers: Everyone
Chicks4FreeID: A Benchmark Dataset for Chicken Re-Identification
Daria Kern, Tobias Schiele, Ulrich Klauck, Winfred Ingabire
- 28 May 2024 (modified: 13 Nov 2024)
- Submitted to NeurIPS 2024 Track Datasets and Benchmarks
- Readers: Everyone
II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models
Ziqiang Liu, Feiteng Fang, Xi Feng, Xeron Du, Chenhao Zhang, Noah Wang, yuelin bai, Qixuan Zhao, Liyang Fan, CHENGGUANG GAN, Hongquan Lin, Jiaming Li, Yuansheng Ni, Haihong Wu, Yaswanth Narsupalli, Zhigang Zheng, Chengming Li, Xiping Hu, Ruifeng Xu, Xiaojun Chen et al. (6 additional authors not shown)
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injection
Jiaqi Wang, Xiaochen Wang, Lingjuan Lyu, Jinghui Chen, Fenglong Ma
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Spotlight
- Readers: Everyone
LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K
Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, Yu Wang
- 28 May 2024 (modified: 13 Nov 2024)
- Submitted to NeurIPS 2024 Track Datasets and Benchmarks
- Readers: Everyone
SHDocs: A dataset, benchmark, and method to efficiently generate high-quality, real-world specular highlight data with near-perfect alignment
Jovin Leong, Koa Ming Di, Benjamin Cham Wen Bin, Shaun Heng
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
Mars: Situated Inductive Reasoning in an Open-World Environment
Xiaojuan Tang, Jiaqi Li, Yitao Liang, Song-Chun Zhu, Muhan Zhang, Zilong Zheng
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, Kai Chen
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
AlleNoise - large-scale text classification benchmark dataset with real-world label noise
Alicja Rączkowska, Aleksandra Osowska-Kurczab, Jacek Szczerbiński, Kalina Jasinska-Kobus, Klaudia Nazarko
- 28 May 2024 (modified: 13 Nov 2024)
- Submitted to NeurIPS 2024 Track Datasets and Benchmarks
- Readers: Everyone
Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation
Juan Claude Formanek, Callum Rhys Tilbury, Louise Beyers, Jonathan Phillip Shock, Arnu Pretorius
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
AllClear: A Comprehensive Dataset and Benchmark for Cloud Removal in Satellite Imagery
Hangyu Zhou, Chia Hsiang Kao, Cheng Perng Phoo, Utkarsh Mall, Bharath Hariharan, Kavita Bala
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
OCTrack: Benchmarking the Open-Corpus Multi-Object Tracking
Zekun Qian, Ruize Han, Wei Feng, Junhui Hou, Linqi Song, Song Wang
- 28 May 2024 (modified: 13 Nov 2024)
- Submitted to NeurIPS 2024 Track Datasets and Benchmarks
- Readers: Everyone
OVT-B: A New Large-Scale Benchmark for Open-Vocabulary Multi-Object Tracking
Haiji Liang, Ruize Han
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
ConceptFactory: Facilitate 3D Object Knowledge Annotation with Object Conceptualization
Jianhua Sun, Yuxuan Li, Longfei Xu, Nange Wang, Jiude Wei, Yining Zhang, Cewu Lu
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
WikiDBs: A Large-Scale Corpus Of Relational Databases From Wikidata
Liane Vogel, Jan-Micha Bodensohn, Carsten Binnig
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Spotlight
- Readers: Everyone
Lean Workbook: A large-scale Lean problem set formalized from natural language math problems
Huaiyuan Ying, Zijian Wu, Yihan Geng, JIayu Wang, Dahua Lin, Kai Chen
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
Einsum Benchmark: Enabling the Development of Next-Generation Tensor Execution Engines
Mark Blacher, Christoph Staudt, Julien Klaus, Maurice Wenig, Niklas Merk, Alexander Breuer, Max Engel, Sören Laue, Joachim Giesen
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
A Retrospective on the Robot Air Hockey Challenge: Benchmarking Robust, Reliable, and Safe Learning Techniques for Real-world Robotics
Puze Liu, Jonas Günster, Niklas Funk, Simon Gröger, Dong Chen, Haitham Bou Ammar, Julius Jankowski, Ante Marić, Sylvain Calinon, Andrej Orsula, Miguel Olivares-Mendez, Hongyi Zhou, Rudolf Lioutikov, Gerhard Neumann, Amarildo Likmeta, Amirhossein Zhalehmehrabi, Thomas Bonenfant, Marcello Restelli, Davide Tateo, Ziyuan Liu et al. (1 additional authors not shown)
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
StreamBench: Towards Benchmarking Continuous Improvement of Language Agents
Cheng-Kuang Wu, Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Hung-yi Lee
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models
Arshia Hemmat, Adam Davies, Tom A. Lamb, Jianhao Yuan, Philip Torr, Ashkan Khakzar, Francesco Pinto
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset
Josef Dai, Tianle Chen, Xuyao Wang, Ziran Yang, Taiye Chen, Jiaming Ji, Yaodong Yang
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Poster
- Readers: Everyone
NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA
Marlon Tobaben, Mohamed Ali Souibgui, Rubèn Tito, Khanh Nguyen, Raouf Kerkouche, Kangsoo Jung, Joonas Jälkö, Lei Kang, Andrey Barsky, Vincent Poulain d'Andecy, Aurélie JOSEPH, Aashiq Muhamed, Kevin Kuo, Virginia Smith, Yusuke Yamasaki, Takumi Fukami, Kenta Niwa, Iifan Tyou, Hiro Ishii, Rio Yokota et al. (7 additional authors not shown)
- 28 May 2024 (modified: 13 Nov 2024)
- Submitted to NeurIPS 2024 Track Datasets and Benchmarks
- Readers: Everyone
Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli
Christopher Wang, Adam Uri Yaari, Aaditya K Singh, Vighnesh Subramaniam, Dana Rosenfarb, Jan DeWitt, Pranav Misra, Joseph R. Madsen, Scellig Stone, Gabriel Kreiman, Boris Katz, Ignacio Cases, Andrei Barbu
- Published: 26 Sept 2024, Last Modified: 13 Nov 2024
- NeurIPS 2024 Track Datasets and Benchmarks Oral
- Readers: Everyone