Diffusion Counterfactual Generation with Semantic Abduction

Rajat R Rasal, Avinash Kori, Fabio De Sousa Ribeiro, Tian Xia, Ben Glocker

TL;DR: A general framework that integrates semantic representations into diffusion models through the lens of Pearlian causality for image editing via counterfactual reasoning.

Abstract: Counterfactual image generation presents significant challenges, including preserving identity, maintaining perceptual quality, and ensuring faithfulness to an underlying causal model. While existing auto-encoding frameworks admit semantic latent spaces which can be manipulated for causal control, they struggle with scalability and fidelity. Advancements in diffusion models present opportunities for improving counterfactual image editing, having demonstrated state-of-the-art visual quality, human-aligned perception and representation learning capabilities. Here, we present a suite of diffusion-based causal mechanisms, introducing the notions of spatial, semantic and dynamic abduction. We propose a general framework that integrates semantic representations into diffusion models through the lens of Pearlian causality to edit images via a counterfactual reasoning process. To the best of our knowledge, ours is the first work to consider high-level semantic identity preservation for diffusion counterfactuals and to demonstrate how semantic control enables principled trade-offs between faithful causal control and identity preservation.

Lay Summary:

Counterfactual image editing answers "what if" questions about an image by making precise changes while preserving all other details, e.g. what if this person’s expression changed, they wore sunglasses, or they were older? In each case, we want to modify only the requested attribute while keeping the person’s identity, background, and other unrelated features unchanged.

Our work uses powerful image generation models, called diffusion models, to create counterfactual images. We guide diffusion models so that causally related properties of the image are modified together, i.e. smiling will cause your mouth to open, while unrelated properties remain unchanged. We find that providing a compact summary of the original image, called a semantic embedding, as an input to the diffusion model further improves identity preservation in the counterfactual.

Diffusion models became popular for generating images from random text prompts, as seen in tools like MidJourney and DALL-E. We believe that a counterfactual perspective is essential to adapting diffusion models for broader, real-world applications. Our approach could help diffusion models better generalise to tasks that require creative reasoning and a deeper understanding of cause and effect.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/RajatRasal/Diffusion-Counterfactuals

Primary Area: Deep Learning->Generative Models and Autoencoders

Keywords: counterfactuals, causality, diffusion, image editing

Submission Number: 5216