Bridging Local Divergence in Federated Learning via Personalized Adjustment Matrix

Shupeng Shen, Zirui Gong, Leo Yu Zhang, Jia Wang

Keywords: Personalized Federated Learning; Cross-client divergence; Classifier–feature misalignment

Abstract: Data distribution divergence across clients often leads to misalignment between global federated models and local decision boundaries. While existing personalized federated learning approaches attempt to mitigate this through feature alignment or multi-head personalization, they typically introduce additional communication and local computation overhead, which in turn limits their effectiveness under severe heterogeneity. In this work, we introduce FedPAM, a complementary perspective that addresses this challenge through a client-specific Personalized Adjustment Matrix (PAM) combined with a contrastive alignment objective, achieving robust personalization with minimal additional cost, while keeping the standard FedAvg communication protocol unchanged. Experiments across diverse benchmarks confirm that FedPAM improves upon competitive personalized FL baselines, showing pronounced advantages in highly heterogeneous conditions.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 11742