Towards Cross-Modal Causal Structure and Representation Learning

Haiyi Mao, Hongfu Liu, Jason Xiaotian Dou, Panayiotis V. Benos

Abstract: Does the SARS-CoV-2 virus cause patients’ chest X-Rays ground-glass opacities? Does an IDH-mutation cause differences in patients’ MRI images? Conventional causal discovery algorithms, although well developed to uncover the cause-effect relationships on structured data, cannot elucidate causal relations between unstructured images and structured scalar variables due to the complexity of the former. In this paper, we consider causal discovery between images and structured (scalar) variables. Specifically, we derive low dimensional image representations to analyze with structured variables. We propose a two-module amortized variational algorithm named Cross-Modal Variational Causal representation and structure Learning (CMCL). CMCL jointly learns identifiable representations given a set of independent structured variables and causal relations via formulating latent representations and structured variables into a direct acyclic graph. Moreover, we further enforce counterfactual invariance/variance onto representations. We demonstrate that CMCL outperforms other related methods on synthetic datasets and validate causal relations on semi-synthetic datasets by visualization.