On the Dual Problem of Convexified Convolutional Neural Networks
Abstract: We study the dual problem of convexified convolutional neural networks (DCCNNs). First, we introduce a primal learning problem motivated by convexified convolutional neural networks (CCNNs), and then construct the dual convex training program through careful analysis of the Karush-Kuhn-Tucker (KKT) conditions and Fenchel conjugates. Our approach reduces the computational overhead of constructing a large kernel matrix and more importantly, eliminates the ambiguity of factorizing the matrix. Due to the low-rank structure in CCNNs and the related subdifferential of nuclear norms, there is no closed-form expression to recover the primal solution from the dual solution. To overcome this, we propose a highly novel weight recovery algorithm, which takes the dual solution and the kernel information as the input, and recovers the linear weight and the output of convolutional layer, instead of weight parameter. Furthermore, our recovery algorithm exploits the low-rank structure and imposes a small number of filters indirectly, which reduces the parameter size. As a result, DCCNNs inherit all the statistical benefits of CCNNs, while enjoying a more formal and efficient workflow.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: Dear Action Editor, we have updated our manuscript:
- Page 2 (first bullet), Table 1 (caption), Page 26 (CCNN kernel matrix factorization paragraph), Table 2 (caption): we clarified that the matrix Q is tall.
- Page 3 (Summary of our contribution): we added appropriate references to the third bullet.
- Section 4: We added more details regarding the algorithms and parameters used.
- Appendix E.1: we updated our algorithm and argued for its convergence.
- We made several other typo corrections.
Supplementary Material: pdf
Assigned Action Editor: ~Jeremias_Sulam1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 1508
Loading