Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Learning Deep Models: Critical Points and Local Openness
Maher Nouiehed, Meisam Razaviyayn
Feb 11, 2018 (modified: Feb 15, 2018)ICLR 2018 Workshop Submissionreaders: everyoneShow Bibtex
Abstract:In this paper we present a unifying framework to study the local/global optima equivalence of the optimization problems arising from training non-convex deep models. Using the local openness property of the underlying training models, we provide simple sufficient conditions under which any local optimum of the resulting optimization problem is globally optimal. We first completely characterize the local openness of matrix multiplication mapping in its range. Then we use our characterization to: 1) show that every local optimum of two layer linear networks is globally optimal. Unlike many existing results, our result requires no assumption on the target data matrix Y, and input data matrix X. 2) Develop almost complete characterization of the local/global optima equivalence of multi-layer linear neural networks. 3) Show global/local optima equivalence of non-linear deep models having certain pyramidal structure. Unlike some existing works, our result requires no assumption on the differentiability of the activation functions.
Keywords:Non-convex Optimization, Local/Global Equivalence, Training Deep Models, Local Open Mappings
Enter your feedback below and we'll get back to you as soon as possible.