Evaluating the Frontier: A Comparative Analysis of Leading LLMs in Advanced Cognitive Tasks

Anonymous

TL;DR: The paper provides a comparative analysis of mainstream leading LLMs, highlighting their capabilities and limitations in advanced cognitive

Abstract:

In recent advancements of natural language processing, Large Language Models (LLMs) have demonstrated unprecedented capabilities in understanding, generating, and interacting with text-based data.
This evaluation explores the ability of proficiency in mainstream LLMs including GPT-3.5, GPT-4.0-turbo, GPT-4.0-vision-preview,Claude 2, and Gemini Pro, extending across mathematics, implicit reasoning, long-context understanding, multi-modal reasoning , and fault-identification abilities.
Current literature often underscores the qualitative triumphs of LLMs without quantifying their holistic abilities and limitations in rigorous scenarios.
This study aims to fill this gap through a series of methodically crafted evaluations.
Our methodology for assessing the capabilities of mainstream Large Language Models (LLMs) is grounded in a hands-on approach, leveraging the practical functionality of their respective API endpoints. Through this method, we engage in a stringent analysis of their performance, ultimately quantifying their effectiveness using a reliable scores—metrics system that constitute a balanced representation of each model's capabilities.And we've crafted unique prompts and experimental designs for different tasks to test each model's strengths and weaknesses effectively.

Paper Type: long

Research Area: Resources and Evaluation

Contribution Types: NLP engineering experiment, Data resources, Data analysis, Surveys

Languages Studied: English