Revisiting the Goldilocks Zone in Inhomogeneous Networks

Zacharie Garnier Cuchet, Sarath Chandar, Ekaterina Lobacheva

Keywords: Goldilocks Zone, Initialization, Trainability, Curvature Analysis, Architectural Inhomogeneity

TL;DR: We analyze the Goldilocks zone in inhomogeneous networks and show that softmax temperature scaling provides a more robust probe of curvature and trainability than weight scaling.

Abstract:

We investigate how architectural inhomogeneities, such as biases, layer normalization, and residual connections, affect the curvature of the loss landscape at initialization and its link to trainability. We focus on the Goldilocks zone, a region in parameter space with excess positive curvature, previously associated with improved optimization in homogeneous networks. To extend this analysis, we compare two scaling strategies: weight scaling and softmax temperature scaling.
Our results show that in networks with biases or residual connections, both strategies identify a Goldilocks zone aligned with better training. In contrast, layer normalization leads to lower or negative curvature, yet stable optimization, revealing a disconnect between curvature and trainability. Softmax temperature scaling behaves more consistently across models, making it a more robust probe. Overall, the Goldilocks zone remains relevant in inhomogeneous networks, but its geometry and predictive power depend on architectural choices, particularly normalization.

Student Paper: Yes

Submission Number: 61