Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track

Rylan Schaeffer, Joshua Kazdan, Yegor Denisov-Blanch, Brando Miranda, Matthias Gerstgrasser, Susan Zhang, Andreas Haupt, Isha Gupta, Elyas Obbad, Jesse Dodge, Jessica Zosa Forde, Francesco Orabona, Sanmi Koyejo, David L. Donoho

Keywords: peer review, machine learning, machine learning conferences, reform, new track, peer review reform

TL;DR: ML conferences should establish a "refutations and critiques" track

Abstract:

Science progresses by iteratively advancing and correcting humanity's understanding of the world. In machine learning (ML) research, rapid advancements have led to an explosion of publications, but have also led to misleading, incorrect, flawed or perhaps even fraudulent studies being accepted and sometimes highlighted at ML conferences due to the fallibility of peer review. While such mistakes are understandable, ML conferences do not offer robust processes to help the field systematically correct when such errors are made.
This position paper argues that ML conferences should establish a dedicated "Refutations and Critiques" (R&C) Track. This R&C Track would provide a high-profile, reputable platform to support vital research that critically challenges prior research, thereby fostering a dynamic self-correcting research ecosystem.
We discuss key considerations including track design, review principles, potential pitfalls, and provide an illustrative example submission concerning a recent ICLR 2025 Oral.
We conclude that ML conferences should create official, reputable mechanisms to help ML research self-correct.

Lay Summary:

Machine learning conferences sometimes accept papers that later turn out to be misleading, flawed, or even wrong. Today, there is no reliable, high-visibility way to correct the scientific record once this happens, which can lead to misguided future research. Critiques are often left to informal channels like social media, which lack scientific evaluation and visibility.

This paper proposes a dedicated "Refutations and Critiques" track at major conferences. Such a track would provide a formal, peer-reviewed venue for scientists to publish papers that rigorously identify, analyze, and correct significant flaws in prior work.

The track would make machine learning more self-correcting as a field of academic study. It would reduce wasted effort, raise research standards, and improve public trust.

Submission Number: 492