A Quotient Homology Theory of Representation in Neural Networks

Kosio Beshkov

Abstract:

Previous research has proven that the set of maps implemented by neural networks with a ReLU activation function is identical to the set of piecewise linear continuous maps. Furthermore, such networks induce a hyperplane arrangement splitting the input domain of the network into convex polyhedra $G_J$ over which a network $\Phi$ operates in an affine manner.

In this work, we leverage these properties to define an equivalence relation $\sim_\Phi$ on top of an input dataset, which defines a quotient space that can be split into two sets related to the local rank of $\Phi_J$ and the intersections $\cap \text{Im}\Phi_{J_i}$. We refer to the latter as the \textit{overlap decomposition} $\mathcal{O}_\Phi$ and prove that if the intersections between each polyhedron and an input manifold are convex, the homology groups of neural representations are isomorphic to quotient homology groups $H_k(\Phi(\mathcal{M})) \simeq H_k(\mathcal{M}/\mathcal{O}_\Phi)$. This lets us intrinsically calculate the Betti numbers of neural representations without the choice of an external metric. We develop methods to numerically compute the overlap decomposition through linear programming and a union-find algorithm.

Submission Type: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

All changes to the manuscript are highlighted in **blue**. The responses to all reviewers also explicitly refer to the changes. Please see the attached pdf for the full revision. Below we provide a short summary of the main changes.

- Following the recommendation from reviewer EvaN, we have clarified the mathematical notation by explicitly defining the equivalence relation and the quotient space. We now also avoid talking about $\sim_\Phi$ as a set which caused confusion.
- We have added a "Conclusion" section, which summarizes what we have done in the paper and explicitly names future directions for research.
- We have extended the discussion of "Topological versus geometric transformations" to address the problem of generalization versus memorization.
- We have improved the "Prerequisites" section by providing some missing definitions.
- We have added a section in the Appendix C.2 which shows that our method generalizes to other architectures such as CNNs, RNNs and ResNets. For ResNets we provide a derrivation of the locally affine expressions that can be used to calculate the Overlap decomposition.
- We have performed simulations to study the false negative rate as a function of the $\delta$ parameter as suggested by reviewer GyBG. We also added a subsection in the Appendix D.5 that describes that process along with Figure 11. which shows the results.

Code: https://github.com/KBeshkov/QuotientHomology

Assigned Action Editor: ~Tianshu_Yu2

Submission Number: 7613