InfoNet: An Efficient Feed-Forward Neural Estimator for Mutual Information
Primary Area: metric learning, kernel learning, and sparse coding
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Deep Learning, Efficient Mutual Information Estimation, Real-Time Correlation Computation, Maximum Correlation Coefficient
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Estimating mutual correlations between random variables or data streams is crucial for intelligent behavior and decision-making. As a fundamental quantity for measuring statistical relationships, mutual information has been widely studied and used for its generality and equitability. However, existing methods either lack the efficiency required for real-time applications or the differentiability necessary with end-to-end learning frameworks. In this paper, we present InfoNet, a feed-forward neural estimator for mutual information that leverages the attention mechanism and the computational efficiency of deep learning infrastructures. By training InfoNet to maximize a dual formulation of mutual information via a feed-forward prediction, our approach circumvents the time-consuming test-time optimization and comes with the capability to avoid local minima in gradient descent. We evaluate the effectiveness of our proposed scheme on various families of distributions and check its generalization to another important correlation metric, i.e., the Hirschfeld-Gebelein-Rényi Maximum Correlation Coefficient. Our results demonstrate a graceful efficiency-accuracy trade-off and order-preserving properties of InfoNet, providing a comprehensive toolbox for estimating both the Shannon Mutual Information and the HGR Correlation Coefficient. We will make the code and trained models publicly available and hope it can facilitate studies in different fields that require real-time mutual correlation estimation.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4727
Loading