Input Invex Neural Network
Abstract: Connected decision boundaries are used in different areas like image segmentation, clustering, alpha-shape or defining a region in nD-space. However, methods for generating such connected decision boundaries using neural networks are lacking in the machine learning literature. While exploring such methods, we found that such decision boundaries can be generated by thresholding a special kind of function called an invex function. We find a connection between invex functions and the connectedness of regions and manifolds, and we apply the connectedness and locality as a foundation for interpreting the nD-data-space. In this paper, we present two methods for constructing invex function using neural networks. The first one is based on intuitions developed visually and constraining the function using our method (Gradient Clipped-Gradient Penalty). The second one is based on later findings on the relationship of invex function to the composition of invertible and convex functions. Using connectedness as a basic interpretation method, we create connected region based classifiers. We show that multiple connected set based classifiers can approximate any classification function. In the experiments section, we first use the invex function for regression and classification tasks to visualize the global optimality and connected set in 2D toy datasets. Furthermore, we use our methods for classification tasks using an ensemble of models as well as using a single model on larger-scale datasets. The experiments show that connected set based classifiers do not have a significant disadvantage over ordinary neural network classifiers. We also evaluate various properties of invex function and connected sets. The overall exploration of this work suggests that invex function is fundamental to understanding and applying locality and connectedness of input space which is useful for multiple tasks.
Submission Length: Long submission (more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=H2PuZ5afos
Changes Since Last Submission: Main change: Revision following suggestions from reviewers.
1. Modification of Introduction section to clarify the contents and motivation.
2. Removal of irrelevant contents and clarification of unclear statements, and moving of contents to and from Appendix.
3. Addition of methodology for interpretation.
4. Addition of experiments for interpretation and network morphism.
5. Simple bug fix to iResNet, resulting in change in experiment settings and output data, however the observation remain same.
6. The changes has been made to overall content to clarify and accommodate suggestions
Major changes in text or topics highlighted with blue color.
Assigned Action Editor: ~Alessandro_Sperduti1
Submission Number: 416
Loading