Battle of the Wordsmiths: Comparing ChatGPT, GPT-4, Claude, and Bard

ICLR 2025 Conference Submission5384 Authors

Keywords: Large language models, ChatGPT, GPT-4, Claude, Bard, datasets, natural language processing, language modeling

TL;DR: We compare four state of the art LLM models as well as introduce a large scale carefully annotated data.

Abstract:

Although informal evaluations of modern LLMs
can be found on social media, blogs, and news
outlets, a formal and comprehensive comparison among them has yet to be conducted. In
response to this gap, we have undertaken an extensive benchmark evaluation of LLMs and conversational bots. Our evaluation involved the collection of 1002 questions encompassing 27 categories,
which we refer to as the “Wordsmiths dataset.”
These categories include reasoning, logic, facts,
coding, bias, language, humor, and more. Each
question in the dataset is accompanied by an accurate and verified answer. We meticulously assessed four leading chatbots: ChatGPT, GPT-4, Bard, and Claude, using this dataset. The results
of our evaluation revealed the following key findings: a) GPT-4 emerged as the top-performing
chatbot across almost all categories, achieving a
success rate of 84.1%. On the other hand, Bard
faced challenges and achieved a success rate of
62.4%. b) Among the four models evaluated,
one of them responded correctly approximately
93% of the time. However, all models were correct only about 44%. c) Bard is less correlated
with other models while ChatGPT and GPT-4 are
highly correlated in terms of their responses. d)
Chatbots demonstrated proficiency in language
understanding, facts, and self-awareness. However, they encountered difficulties in areas such
as math, coding, IQ, and reasoning. e) In terms of
bias, discrimination, and ethics categories, models
generally performed well, suggesting they are relatively safe to utilize. To make future model evaluations on our dataset easier, we also provide a multiple-choice version of it (called WordsmithsMCQ).

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5384