Why every GBM speed benchmark is wrong

Anna Veronika Dorogush, Vasily Ershov, Dmitriy Kruchinin

Keywords: GBM, benchmarks, xgboost, catboost, ligthgbm, boosting

TL;DR: Why every GBM speed benchmark is wrong?

Abstract: This pdf is a proposal for a small talk about speed benchmarks of gradient boosted decision libraries on MLoss 2018 NIPS workshop. There is no proper way to do speed benchmarks of boosting libraries today. Openly-available benchmarks are misleading, often contains mistakes. We want to show, what is wrong with the benchmarks and what everyone should keep in mind, if one want to compare different libraries in a meaningful way.

Decision: accept