Hyper-parameter optimization in classification: To-do or not-to-do

Ngoc Tran, Jean-Guy Schneider, Ingo Weber, A. Kai Qin

Abstract: Highlights•We found hyper-param tuning is not well justified in many cases but still very useful in a few.•We propose a framework to address the problem of deciding to-tune or not-to-tune.•We implemented a prototype of the framework with 486 datasets and 4 algorithm.•The results indicates our framework is effective at avoiding effects of ineffective tuning.•Our framework enables a life-long learning approach to the problem.