MOFA: Modular Factorial Design for Hyperparameter Optimization

Bo Xiong, Yimin Huang, Steffen Staab, Zhenguo Li

Keywords: Automated Hyperparameter Optimization, Factorial Analysis, Model-Free, Sample Efficiency, Orthogonal Latin Hypercubes

Abstract: Automated hyperparameter optimization (HPO) has shown great power in many machine learning applications. While existing methods suffer from model selection, parallelism, or sample efficiency, this paper presents a new HPO method, MOdular FActorial Design (MOFA), to address these issues simultaneously. The major idea is to use techniques from Experimental Designs to improve sample efficiency of model-free methods. Particularly, MOFA runs with four modules in each iteration: (1) an Orthogonal Latin Hypercube (OLH)-based sampler preserving both univariate projection uniformity and orthogonality; (2) a highly parallelized evaluator; (3) a transformer to collapse the OLH performance table into a specified Fractional Factorial Design--Orthogonal Array (OA); (4) an analyzer including Factorial Performance Analysis and Factorial Importance Analysis to narrow down the search space. We theoretically and empirically show that MOFA has great advantages over existing model-based and model-free methods.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: This paper proposed orthogonal Latin hypercube-based multi-module designs and matched factorial analysis, named MOFA, to improve the sample efficiency of model-free methods for hyperparameter optimization.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2011.09545/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=C8w2CnNxTP