SSL Framework for Causal Inconsistency between Structures and Representations

Hang Chen, Xinyu Yang, Keqing Du

Primary Area: causal reasoning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Self-supervised Learning, Causal Representations, Causal Consistency, Interventions

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: To settle down the causal inconsistency between structures and representations, we proposed a new insight for intervention, a SSL framework, two implementation examples, and a brand new dataset.

Abstract:

The cross-pollination of deep learning and causal discovery has catalyzed a burgeoning field of research, seeking to elucidate causal relationships within non-statistical data forms like images, videos, and text. Such data, often being named ‘indefinite data', exhibit unique challenges—inconsistency between causal structure and representation, which are not common in conventional data forms. To tackle this issue, we theoretically develop intervention strategies suitable for indefinite data and derive causal consistency condition (CCC). Moreover, we design a self-supervised learning (SSL) framework that considers interventions as ’views' and CCC as a `philosophy' with two implement examples on Supervised Specialized Models (SSMs) and Large Language Models (LLMs), respectively. To evaluate pure inconsistency manifestations, we have prepared the first high-quality causal dialogue dataset-Causalogue. Evaluations are also performed
on three other downstream tasks. Extensive experimentation has substantiated the efficacy of our methodology, illuminating how CCC could potentially play an influential role in various fields. Our code is available in https://anonymous.4open.science/r/ICLR_Anonymous_submission_575_new_dataset_Causalogue_and_codes/Causalogue_Dataset/Fork_I_demo.py.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 575