Performance of NSGA-III on Multi-objective Combinatorial Optimization Problems Heavily Depends on Its Implementations

Cheng Gong, Yang Nan, Lie Meng Pang, Hisao Ishibuchi, Qingfu Zhang

Abstract: Newly proposed many-objective algorithms have been almost always compared with NSGA-III for performance evaluation. Since the authors of the NSGA-III paper have not provided any source code, researchers usually use an available implementation in popular optimization platforms. This can lead to unreliable comparison results if different performance of NSGA-III is obtained depending on the choice of a platform. In this paper, we show that the implementations of NSGA-III are slightly different between the two most frequently used EMO optimization platforms: PlatEMO and pymoo. Then, we examine the effect of the implementation difference on the performance of NSGA-III in each platform. Our experimental results show that almost the same results are obtained from the two implementations on the frequently-used DTLZ test problems. However, our experimental results also show that clearly different results are obtained from the two implementations on multi-objective combinatorial optimization problems. Finally, we demonstrate that the weaker performance of the PlatEMO implementation of NSGA-III can be improved by replacing its normalization mechanism with the corresponding mechanism in Pymoo. That is, our experimental results show that small differences in the normalization mechanisms of the two implementations lead to large differences in their performance on multi-objective combinatorial optimization problems.