IsoAct: Structure-Preserving Post-hoc Debiasing via Isometric Actions

Sumin Park, Taero Kim, Subeen Park, Minhyeong Cho, Jong Hwan Lee, Kyungwoo Song

Keywords: Trustworthy AI; Fairness and Debiasing; Nuisance-Robust Representations; Post-hoc Concept Erasure; Manifold-aware Representation Learning

Abstract: Learning representations that are robust to nuisance factors is essential for reliable deployment, but remains challenging when the representations lie on non-Euclidean manifolds. Existing debiasing methods typically assume Euclidean representations or apply linear interventions, which can distort the geometry of structured latent spaces. This creates a mismatch between the Euclidean assumptions of the editing operation and the geometric constraints that manifold-valued representations must satisfy. We propose IsoAct, a post-hoc representation-editing framework that reduces nuisance information by composing manifold-native action primitives while maintaining downstream utility and manifold validity. The framework unifies debiasing on hyperspherical, hyperbolic, and SE(3) representations by parameterizing valid transformations within each geometry. We evaluate the approach across synthetic and real-world datasets using debiasing capability, task preservation, and manifold-constraint violation. Across these settings, IsoAct yields a favorable nuisance--utility--validity trade-off relative to Euclidean editing baselines, reducing nuisance recoverability while maintaining competitive task preservation and low manifold-constraint violation.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 320