Is the Deep Model Representation Sparse and Symbolic with Causal Patterns?

Jie Ren, Mingjie Li, Qirui Chen, Huiqi Deng, Quanshi Zhang

Keywords: Representation Learning, Deep Learning Theory, Explainable AI

Abstract: This paper aims to show that the inference logic of a deep model can be faithfully approximated as a sparse, symbolic causal graph. Such a causal graph potentially bridges the gap between connectionism and symbolism. To this end, the faithfulness of the causal graph is theoretically guaranteed, because we show that the causal graph can well mimic the model's output on an exponential number of different masked samples. Besides, such a causal graph can be further simplified and re-written as an And-Or graph (AOG), which explains the logical relationship between interactive concepts encoded by the deep model, without losing much explanation accuracy. The code will be released when the paper is accepted.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Social Aspects of Machine Learning (eg, AI safety, fairness, privacy, interpretability, human-AI interaction, ethics)

TL;DR: This paper shows that the inference logic of a deep model can usually be represented as a sparse causal graph, and the faithfulness of such a symbolic representation is theoretically guaranteed.