CDT: A Comprehensive Capability Framework for Large Language Models Across Cognition, Domain, and Task

ACL ARR 2025 February Submission6331 Authors

Abstract:

Recent advances in Large Language Models (LLMs) have significantly enhanced their capabilities, highlighting the need for comprehensive evaluation frameworks that extend beyond task-specific benchmarks.
However, existing benchmarks often focus on isolated abilities, lacking a holistic framework for assessing LLM capabilities.
To address this gap, we propose the **C**ognition-**D**omain-**T**ask (CDT) framework, which comprehensively measures a model's capabilities across three dimensions.
We expand the scope of model capability definitions at the cognitive level by incorporating the Cattell-Horn-Carroll cognitive theory, refining the categorization of model capabilities.
In addition, we propose two data selection methods based on this framework, which has shown significant improvements in both general and specific benchmarks. These results demonstrate the effectiveness of our CDT framework and its practical utility. Source code and model will be available at https://anonymous.4open.science/r/CDT-641F.

Paper Type: Long

Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics

Research Area Keywords: cognitive modeling

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 6331