Tractable structured natural-gradient descent using local parameterizations

Wu Lin, Frank Nielsen, Mohammad Emtiyaz Khan, Mark Schmidt

Abstract: Natural-gradient descent (NGD) on structured parameter spaces (e.g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations. We address this issue by using...