Go to ACL ARR 2025 February homepageMCQFormatBench: Robustness Tests for Multiple-Choice Questions
Abstract: Multiple-choice questions (MCQs) are often used to evaluate large language models (LLMs). They measure LLMs' general common sense and reasoning abilities, as well as their knowledge in specific domains such as law and medicine. However, the robustness of LLMs to various question formats in MCQs has not been thoroughly evaluated. While there are studies on the sensitivity of LLMs to input variations, research into their responsiveness to different question formats is still limited. Therefore, in this study, we propose a method to construct tasks to comprehensively evaluate the robustness against format changes of MCQs by decomposing the answering process into several steps. Using this dataset, we evaluate eight LLMs, such as Llama3-70B and Mixtral-8x7B. We find the lack of robustness to differences in the format of MCQs. It is crucial to consider whether the format of MCQs influences their evaluation scores when assessing LLMs using MCQ datasets.
Paper Type: Long
Research Area: Language Modeling
Research Area Keywords: Evaluation, MCQ, LLM, Robustness
Contribution Types: Data resources, Data analysis
Languages Studied: English
Submission Number: 7948
Loading