David Heineman

Researcher, Allen Institute for Artificial Intelligence

Joined

June 2023

Names

David Heineman (Preferred)

Emails

****@gatech.edu (Confirmed)

****@allenai.org (Confirmed)

****@gmail.com (Confirmed)

****@stanford.edu (Confirmed)

Personal Links

Career & Education History

Researcher

Allen Institute for Artificial Intelligence (allenai.org)

2024 – 2026

Undergrad student

Georgia Institute of Technology (gatech.edu)

2020 – 2024

Advisors, Relations & Conflicts

Coauthor

Kyle Lo

2025 – 2026

Coauthor

Jesse Dodge

2024 – 2025

Undergraduate Advisor

Wei Xu

2021 – 2024

Expertise

Language Models

2022 – Present

Resources and Evaluation

2022 – Present

Publications

Olmix: A Framework for Data Mixing Throughout LM Development
Mayee F Chen, Tyler Murray, David Heineman, Matt Jordan, Hannaneh Hajishirzi, Christopher Re, Luca Soldaini, Kyle Lo
- ICLR 2026 Workshop DATA-FM Oral
- Readers: Everyone
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mike A Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen et al. (64 additional authors not shown)
- ICLR 2026 Poster
- Readers: Everyone
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
David Heineman, Valentin Hofmann, Ian Magnusson, Yuling Gu, Noah A. Smith, Hannaneh Hajishirzi, Kyle Lo, Jesse Dodge
- NeurIPS 2025 spotlight
- Readers: Everyone
Fluid Language Model Benchmarking
Valentin Hofmann, David Heineman, Ian Magnusson, Kyle Lo, Jesse Dodge, Maarten Sap, Pang Wei Koh, Chun Wang, Hannaneh Hajishirzi, Noah A. Smith
- COLM 2025
- Readers: Everyone
2 OLMo 2 Furious (COLM’s Version)
Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerquin et al. (22 additional authors not shown)
- COLM 2025
- Readers: Everyone
Evaluating LLMs on Chinese Idiom Translation
Cai Yang, Yao Dou, David Heineman, Xiaofeng Wu, Wei Xu
- COLM 2025
- Readers: Everyone
Establishing Task Scaling Laws via Compute-Efficient Model Ladders
Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A. Smith, Dirk Groeneveld, Pang Wei Koh, Jesse Dodge, Hannaneh Hajishirzi
- COLM 2025
- Readers: Everyone
DataDecide: How to Predict Best Pretraining Data with Small Experiments
Ian Magnusson, Nguyen Tai, Ben Bogin, David Heineman, Jena D. Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A. Smith, Pang Wei Koh, Jesse Dodge
- ICML 2025 poster
- Readers: Everyone
Towards a Path Dependent Account of Category Fluency
David Heineman, Reba Koenen, Sashank Varma
- CoRR 2024
- Readers: Everyone
Improving Minimum Bayes Risk Decoding with Multi-Prompt
David Heineman, Yao Dou, Wei Xu
- EMNLP 2024
- Readers: Everyone

View all 16 publications

Co-Authors

View all 143 co-authors