NeurIPS 2025 Workshop LLM Evaluation Submissions

Automated Capability Evaluation of Foundation Models
Arash Afkanpour, Omkar Dige, Fatemeh Tavakoli
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Precursors, Proxies, and Predictive Models for Long-Horizon Tasks
Samuel F. Brown, Jaco Du Toit, Leo Hyams, Daniil Anisimov
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
MiSCHiEF: A Benchmark in Minimal-Pairs of Safety and Culture for Holistic Evaluation of Fine-Grained Image-Caption Alignment
Sagarika Banerjee, Tangatar Madi, Advait Swaminathan, Nguyen Dao Minh Anh, Shivank Garg, Kevin Zhu, Vasu Sharma
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models
Aashiq Muhamed, Leonardo F. R. Ribeiro, Markus Dreyer, Virginia Smith, Mona T. Diab
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attribution
Adi Banerjee, Anirudh Nair, Tarik Borogovac
- Published: 24 Sept 2025, Last Modified: 16 Oct 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
İbrahim Ethem Deveci, Duygu Ataman
- Published: 24 Sept 2025, Last Modified: 01 Nov 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Rethinking MCQ Benchmarks: Mandatory Reasoning Evaluation Reveals Significant Performance Drops in Large Language Models
Yue Zhang, Nhan Nguyen
- Published: 24 Sept 2025, Last Modified: 21 Oct 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
OPTiCAL: An Abstract Positional Reasoning Benchmark for Vision Language Models
Christopher Driggers-Ellis, Gabriel Ayoubi, Christan Grant
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
"It Doesn’t Know Anything About my Work": Participatory Benchmarking and AI Evaluation in Applied Settings
Elizabeth Anne Watkins, Emanuel Moss, Ramesh Manuvinakurike, Christopher Persaud, Giuseppe Raffa, Lama Nachman
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
MEAL: A Multi-dimensional Evaluation of Alignment Techniques for LLMs
Muneeza Azmat, Momin Abbas, Maysa Macedo, Marcelo Carpinette Grave, Luan Soares de Souza, Tiago Lemos de Araujo Machado, Rogério Abreu de Paula, Raya Horesh, Yixin Chen, Heloisa Candello, Rebecka Nordenlöw, Aminat Adebiyi
- Published: 24 Sept 2025, Last Modified: 20 Oct 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
PEBBLE: A Pedagogical and SRL-Aware Benchmark for Evaluating LLM Tutors
Ishaan Gangwani, Harrish Ayyanar, Arjun Rawal
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning
Ziye Chen, Chengwei Qin, Yao Shu
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
No-Human in the Loop: Agentic Evaluation at Scale for Recommendation
Tao Zhang, Kehui Yao, Luyi Ma, Reza Yousefi Maragheh, Jiao Chen, Kai Zhao, Jianpeng Xu, Evren Korpeoglu, Sushant Kumar, Kannan Achan
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
SWE-InfraBench: Evaluating Language Models on Cloud Infrastructure Code
Natalia Tarasova, Enrique Balp-Straffon, Aleksei Iancheruk, Yevhenii Sielskyi, Nikita Kozodoi, Liam H. Byrne, Jack Butler, Dayuan jiang, Marcin Czelej, Andrew Ang, Yash Shah, Roi Blanco, Sergei Ivanov
- Published: 24 Sept 2025, Last Modified: 14 Oct 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Justice in Judgment: Unveiling (Hidden) Bias in LLM-Assisted Peer Reviews
Sai Suresh Macharla Vasu, Ivaxi Sheth, Hui-Po Wang, Ruta Binkyte, Mario Fritz
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Active Model Selection for Large Language Models
Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel
- Published: 24 Sept 2025, Last Modified: 28 Oct 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Detecting Data Contamination in LLMs via In-Context Learning
Michał Zawalski, Meriem Boubdir, Klaudia Bałazy, Besmira Nushi, Pablo Ribalta
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
From Bias to Balance How Multilingual Dataset Composition Affects Tokenizer Performance Across Languages
Aishwarya Selvamurugan, Raj Dandekar, Rajat Dandekar, Sreedath Panat
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction
Huanxin Sheng, Xinyi Liu, Hangfeng He, Jieyu Zhao, Jian Kang
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Human-Centric Framework for Large Multimodal Models Evaluation
Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie, Ashmal Vayani, Mukund Sayeeganesh Chettiar, Deval Pandya
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Talking with Oompa Loompas: A novel framework for evaluating linguistic acquisition of LLMs
Sankalp Tattwadarshi Swain, Anshika Krishnatray, Dhruv Kumar, Jagat Sesh Challa
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Towards Multilingual Mechanistic Interpretability
Yanan Long
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
AgentCaster: Reasoning-Guided Tornado Forecasting
Michael Chen
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
MonitorLLM: Real-Time Structural and Bias Evaluation of Generative AI through Knowledge Graphs
Mohd Ariful Haque, kishor datta gupta, Roy George
- Published: 24 Sept 2025, Last Modified: 24 Sept 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone
Rethinking Kernel Program Repair: Benchmarking and Enhancing LLMs with RGym
Kareem Shehada, Yifan Wu, Wyatt D. Feng, Adithya Iyer, Gryphon Kumfert, Yangruibo Ding, Zhiyun Qian
- Published: 24 Sept 2025, Last Modified: 26 Oct 2025
- NeurIPS 2025 LLM Evaluation Workshop Poster
- Readers: Everyone