Entropy Voting Between Capsules

TMLR Paper4352 Authors

Abstract: Capsule networks offer a promising solution in computer vision by addressing the limitations of convolutional neural networks (CNNs), such as data dependency and viewpoint challenges. Unlike CNNs, capsules reduce the need for data augmentation by enhancing generalization from limited training data. We explore capsules from the perspective of information theory, viewing them as continuous random variables. We use marginal differential entropy to measure the information content of capsules, and relative entropy to model the agreement between lower-level and higher-level capsules. The proposed entropy voting method aims to maximize capsule marginal entropies and to minimize their relative entropy. We show through an ablation study that such a relationship exists between the capsules. We also show that our approach performs better or comparably against state-of-the-art capsule networks while significantly improving inference time. This research highlights the synergy between capsules and information theory, providing insights into their combined potential.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

Clarification of formulae and notation, included "REM: Routing Entropy Minimization for Capsule Networks" by Renzulli et al., clarified Figure 2.

Assigned Action Editor: Georgios Leontidis

Submission Number: 4352