$\texttt{CLR-Bench}$: Evaluating Large Language Models in College-Level Reasoning
Keywords: Large Language Models Evaluation, Benchark and dataset, College-level Reasoning
Abstract: Large language models (LLMs) have demonstrated their remarkable performance across various language understanding tasks. While emerging benchmarks have been proposed to evaluate LLMs in various domains such as mathematics and computer science, they merely measure the accuracy in terms of the final prediction on multi-choice questions. However, it remains insufficient to verify the essential understanding of LLMs given a chosen choice. To fill this gap, we present $\texttt{CLR-Bench}$ to comprehensively evaluate the LLMs in complex college-level reasoning. Specifically, $(i)$ we prioritize 16 challenging college disciplines in computer science and artificial intelligence. The dataset contains 5 types of questions, while each question is associated with detailed explanations from experts. $(ii)$ To quantify a fair evaluation of LLMs' reasoning ability, we formalize the criteria with two novel metrics. Q$\rightarrow$A is utilized to measure the performance of direct **a**nswer prediction, and Q$\rightarrow$AR effectively considers the joint ability to **a**nswer the question and provide **r**ationale simultaneously. Extensive experiments are conducted with 40 LLMs over 1,018 discipline-specific questions. The results demonstrate the key insights that LLMs, even the best closed-source LLM, i.e., GPT-4 turbo, tends to '***guess***' the college-level answers. It shows a dramatic decrease in accuracy from 63.31\% Q$\rightarrow$A to 39.00\% Q$\rightarrow$AR, indicating an unsatisfactory reasoning ability.
Primary Area: datasets and benchmarks
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5453
Loading