Visual Counterfactual Explanations with Compositional Generative Models

Daniil Kirilenko, Dario Fenoglio, Martin Gjoreski, Marc Langheinrich

Keywords: counterfactuals, explanations, compositionality, diffusion

Abstract: Generating realistic counterfactual explanations for vision models requires changing the right visual factors while preserving the rest of the image. Existing approaches typically edit pixels or holistic latent codes, often producing entangled changes that obscure what drove the prediction shift. We propose VOCCE, an object-centric counterfactual generation method that operates on slot-based representations and a slot-conditioned diffusion backbone. Given an input image and a target label, VOCCE steers reverse diffusion using classifier guidance on the denoising prediction and gradient-based updates to the object slots, enabling targeted object- or part-level interventions. To prevent off-manifold slot drift and preserve realism, we introduce a Gaussian Mixture Model prior over slot states as a regularizer. Experiments on ClevrTex and CelebA-HQ, plus cross-dataset transfer from FFHQ to CelebA-HQ show that VOCCE improves locality, closeness, and visual fidelity compared to state-of-the-art counterfactual baselines, and a user study indicates preferences for more meaningful and subtle edits.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 110