Can Models Help us Create Better Models? Evaluating LLMs as Data Scientists

ICLR 2025 Conference Submission12464 Authors

Keywords: llm, data science, benchmark, tabular data, feature engineering, dataset, kaggle, evaluation, code generation

TL;DR: A new benchmark designed to evaluate LLMs on the task of feature engineering, focusing on how well they can apply domain knowledge and generate effective code for tabular datasets.

Abstract:

We present a benchmark for large language models
designed to tackle one of the most knowledge-intensive tasks in data science: writing
\textit{feature engineering} code, which requires domain knowledge in addition to a deep understanding of the underlying problem and data structure.
The model is provided with a dataset description in a prompt and asked to generate code transforming it. The evaluation score is derived from the improvement achieved by an XGBoost model fit on the modified
dataset compared to the original data. By an extensive evaluation of state-of-the-art models and comparison to well-established benchmarks, we demonstrate that the \bench{} of our proposal can cheaply and efficiently assess the broad capabilities of LLMs, in contrast to the existing methods.
The reference implementation is available at \url{ATTACHED_AS_SUPPLEMENTARY}

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12464