Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
USING FUNCTION SPACE THEORY FOR UNDERSTANDING INTERMEDIATE LAYERS
Oren Elisha, Shai Dekel
Jan 29, 2018 (modified: Jan 29, 2018)ICLR 2018 Workshop Submissionreaders: everyone
Abstract:The representational change of input along the intermediate layers is an important
aspect of understanding deep learning architectures. To this end, we propose an
approach that relies on the foundation of Function Space theory. In particular, we
argue that a weak-type Besov smoothness index can quantify the geometry of the
clustering in the feature space of each layer. Therefore, our approach may provide
an additional perspective for understanding data-models fit in the setting of deep
learning. While using a different framework and perspective, the experiments we
performed are in line with the results described by Tishby & Zaslavsky (2015) and
Montavon et al. (2010) in the sense that for well-performing trained networks, the
quality of the representation increases from layer to layer. Our approach could
also be used for addressing generalization (Zhang et al., 2016), (Kawaguchi et al.,
2017) as we also show that the Besov smoothness of the layer representations of
the training set decreases as we add more mis-labeling.
TL;DR:We propose a Function Space theory approach, that describes the change of the input along the intermediate layers in deep learning architectures
Keywords:deep learning, representation layers, Function Space, wavelets, approximation, Besov smoothness
Enter your feedback below and we'll get back to you as soon as possible.