Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU ActivationsDownload PDF

21 May 2021, 20:45 (edited 26 Oct 2021)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: learning theory, polynomial time algorithms, learning ReLU networks, tensor decompositions
  • TL;DR: We give polynomial time algorithms that provably learn depth-2 neural networks with general ReLU activations (having bias terms) under mild conditions.
  • Abstract: We present polynomial time and sample efficient algorithms for learning an unknown depth-2 feedforward neural network with general ReLU activations, under mild non-degeneracy assumptions. In particular, we consider learning an unknown network of the form $f(x) = {a}^{\mathsf{T}}\sigma({W}^\mathsf{T}x+b)$, where $x$ is drawn from the Gaussian distribution, and $\sigma(t) = \max(t,0)$ is the ReLU activation. Prior works for learning networks with ReLU activations assume that the bias ($b$) is zero. In order to deal with the presence of the bias terms, our proposed algorithm consists of robustly decomposing multiple higher order tensors arising from the Hermite expansion of the function $f(x)$. Using these ideas we also establish identifiability of the network parameters under very mild assumptions.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
13 Replies