Unreflected Use of Tabular Data Repositories Can Undermine Research Quality

Andrej Tschalzev, Lennart Purucker, Stefan Lüdtke, Frank Hutter, Christian Bartelt, Heiner Stuckenschmidt

Keywords: Machine Learning for tabular data, Deep Learning for tabular data, Data-centric AI, Repository, Tabular Data, OpenML

TL;DR: Uncritical use of tabular data repositories compromises research quality through flawed model selection, overlooked baselines, and improper preprocessing, demanding better repository standards.

Abstract: Data repositories have accumulated a large number of tabular datasets from various domains. Machine Learning researchers are actively using these datasets to evaluate novel approaches. Consequently, data repositories have an important standing in tabular data research. They not only host datasets but also provide information on how to use them in supervised learning tasks. In this paper, we argue that, despite great achievements in usability, the unreflected usage of datasets from data repositories may have led to reduced research quality and scientific rigor. We present examples from prominent recent studies that illustrate the problematic use of datasets from OpenML, a large data repository for tabular data. Our illustrations help users of data repositories avoid falling into the traps of (1) overfitting validation data during model selection, (2) overlooking strong baselines, and (3) inappropriate preprocessing. In response, we discuss possible solutions for how data repositories can prevent inappropriate use of datasets and become the cornerstones for improved overall quality of empirical research studies.

Submission Number: 10