Riemannian Geometry for Fairness in Attention Mechanisms of Language Models

Md Nur Amin, Alexander Jesser

Abstract: We propose a method, Curvature-Aware Fairness in Attention (CAFA), for mitigating demographic biases in transformer-based language models. By viewing the attention space as a Riemannian manifold, CAFA adjusts the metric tensor and Ricci curvature to reshape how tokens interact. Our analysis proves that these geometric modifications alter geodesic paths, driving attention away from stereotypical associations. Curvature modifications constrain the spread of biased signals by influencing the manifold's geodesic flow, resulting in a more balanced attention distribution across diverse contexts. Empirical evaluations on the StereoSet benchmark show a notable reduction in stereotype scores, with only a minimal impact on language modeling performance.

External IDs:dblp:conf/semco/AminJ25