Interpretable Estimation of CNN Deep Feature Density using Copula and the Generalized Characteristic Function

TMLR Paper4109 Authors

Abstract: We present a novel empirical approach toward estimating the Probability Density Function (PDF) of the deep features of Convolutional Neural Networks (CNNs). Estimating the PDF of deep CNN features is an important task, because it will yield new insight into deep representations. Moreover, characterizing the statistical behavior has implications for the feasibility of promising downstream tasks such as density based anomaly detection. Expressive, yet interpretable estimation of the deep feature PDF is challenging due to the Curse of Dimensionality (CoD) as well as our limited ability to comprehend high-dimensional inter-dependencies. Our novel estimation technique combines copula analysis with the Method of Orthogonal Moments (MOM), in order to directly estimate the Generalized Characteristic Function (GCF) of the multivariate deep feature PDF. We find that the one-dimensional marginals of non-negative deep CNN features after major blocks are not well approximated by a Gaussian distribution, and that the features of deep layers are much better approximated by the Exponential, Gamma, and/or Weibull distributions. Furthermore, we observe that deep features become increasingly long-tailed with network depth, although surprisingly the rate of this increase is much slower than theoretical estimates. Finally, we observe that many deep features exhibit strong dependence (either correlation or anti-correlation) with other extremely strong detections, even if these features are independent within typical ranges. We elaborate on these findings in our discussion, where we hypothesize that the long-tail of large valued features corresponds to the strongest computer vision detections of semantic targets, which would imply that these large-valued features are not outliers but rather an important detection signal.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=VGy6k9PFr5

Changes Since Last Submission:

1. Greatly Expanded Literature Review (Long-tail theory, Linear decompositions, Feasibility of anomaly detection, Copula / GCF theory)

2. Greatly Expanded Methodology including Formal Definition and Algorithmic Design

3. New Experimental Analysis of Optimal Tail parameter with comparison to theoretical estimates

4. New Intercomparison with related Archimedean copulas and Empirical Characteristic Function methods.

5. Expanded Conclusion and Discussion (Discussion now includes results from optimal tail parameters, Exponential view hypothesis replaced with long-tail view)

6. New Appendices to provide further details for experimental design and algorithmic design

Assigned Action Editor: ~Murat_A_Erdogdu1

Submission Number: 4109