Characterising Deep Learning Loss Landscapes with Local Optima Networks

Yuyang Zhou, Ferrante Neri, Ruibin Bai

Abstract: Deep learning has gained significant popularity in recent years, particularly for tasks like image and speech recognition, natural language processing, and other intricate pattern recognition challenges. However, training a deep learning model involves tuning millions or even billions of parameters. Consequently, this training process becomes a large-scale optimisation problem associated with a mostly unknown but highly non-convex fitness landscape. In recent decades, advances in fitness landscape analysis have revolved around characterizing landscapes representing loss functions, with Local Optima Networks (LONs) emerging as a promising tool. This paper, while focusing on LeNet-5, leverages LON to address four key questions concerning the nature of the learning problem. We emphasize the impact of experimental conditions during the analysis phase on drawing conclusions about the problem's nature. The results shed light on parametrization and optimiser selection to enhance the analysis and comprehension of deep learning loss landscapes. In particular, we identify the presence and number of funnels in the landscape's structure, study the impact of the dataset on the nature of the problem, investigate how the choice of local search optimisers may influence conclusions about the problem's structure. Finally, sensitivity analysis was conducted on the perturbation strength of the Basin-Hopping sampling method for LON construction.