Abstract: Highlights•Novel adversarial contrastive FL framework for domain-invariant knowledge learning.•Prototype contrastive learning and supervised contrastive learning method for client–server adversarial training.•The Gaussian Mixture Model is introduced to the aggregation strategy.•Achieving communication efficiency, privacy, and comparable performance to existing FL.