Dieuwke Hupkes

Research Scientist, Facebook

Joined

February 2019

Names

Dieuwke Hupkes (Preferred)

Emails

****@uva.nl (Confirmed)

****@gmail.com (Confirmed)

****@fb.com (Confirmed)

****@meta.com (Confirmed)

Personal Links

Homepage

Google Scholar

DBLP

Career & Education History

Research Scientist

Facebook (facebook.com)

2020 – Present

Postdoc

University of Amsterdam (uva.nl)

2020 – 2020

PhD student

University of Amsterdam (uva.nl)

2015 – 2020

Advisors, Relations & Conflicts

No relations added

Expertise

Natural Language Processing

Present

RNNs

Present

Compositionality

Present

Generalisation

Present

Publications

MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages
Dieuwke Hupkes, Nikolay Bogoychev
- Submitted to NeurIPS 2025 Datasets and Benchmarks Track
- Readers: Everyone
MLGym: A New Framework and Benchmark for Advancing AI Research Agents
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Mikhail Plekhanov, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Nicolaus Foerster, Yoram Bachrach, William Yang Wang, Roberta Raileanu
- COLM 2025
- Readers: Everyone
Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks
Rylan Schaeffer, Punit Singh Koura, Binh Tang, Ranjan Subramanian, Aaditya K Singh, Todor Mihaylov, Prajjwal Bhargava, Lovish Madaan, Niladri S. Chatterji, Vedanuj Goswami, Sergey Edunov, Dieuwke Hupkes, Sanmi Koyejo, Sharan Narang
- Submitted to ICLR 2025
- Readers: Everyone
Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, Dieuwke Hupkes
- ICLR 2025 Conference Withdrawn Submission
- Readers: Everyone
Quantifying Variance in Evaluation Benchmarks
Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, Dieuwke Hupkes
- Submitted to ICLR 2025
- Readers: Everyone
Quantifying Variance in Evaluation Benchmarks
Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, Dieuwke Hupkes
- RegML 2024
- Readers: Everyone
The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks
Kaiser Sun, Adina Williams, Dieuwke Hupkes
- CoNLL2023
- Readers: Everyone
Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation
Verna Dankers, Ivan Titov, Dieuwke Hupkes
- EMNLP 2023 Main
- Readers: Everyone
A Replication Study of Compositional Generalization Works on Semantic Parsing
Kaiser Sun, Adina Williams, Dieuwke Hupkes
- Published: 02 Aug 2023, Last Modified: 02 Aug 2023
- MLRC 2022 OutstandingPaper
- Readers: Everyone
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang et al. (430 additional authors not shown)
- Accepted by TMLR
- Readers: Everyone

View all 20 publications

Co-Authors

View all 505 co-authors