Exploring Dimensionality Reduction Techniques for Efficient Surrogate-Assisted optimization

Sibghat Ullah, Duc Anh Nguyen, Hao Wang, Stefan Menzel, Bernhard Sendhoff, Thomas Bäck

Abstract: Constructing surrogate models of high dimensional optimization problems is challenging due to the computational complexity involved. This paper empirically investigates the practicality of major dimensionality reduction techniques for encapsulating the high dimensional design space into compact representations. Such low dimensional representations of the design space can be utilized for constructing the surrogate models efficiently. Based on historical mainstays and recent developments in deep learning, we study four dimensionality reduction techniques in this paper, namely Principal Component Analysis, Kernel Principal Component Analysis, Autoencoders and Variational Autoencoders. We evaluate and compare these techniques based on quality assessments of the corresponding low dimensional surrogate models on a diverse range of test cases. These test cases are defined on combinations of three dimensionsalities, ten well-known benchmark problems from the continuous optimization domain and two surrogate modeling techniques, namely Kriging and Polynomials. Our results clearly demonstrate the superiority of Autoencoders and Principal Component Analysis on the criteria of modeling accuracy and global optimality respectively.