Metadata-Agnostic Decentralized Learning

Tianyu Zhang, Lu Li, Tongtian Zhu, Suyuchen Wang, Can Wang, Yong Chen

Keywords: Decentralized Learning, Metadata-Agnostic

TL;DR: We propose to use metadata-agnostic approach in decentralized learning for more realistic evaluations. We emphasis algorithms can get very different conclusion based on different evaluation settings.

Abstract: Decentralized learning enables collaborative model training without sharing raw data, offering strong privacy benefits. However, many existing studies in decentralized learning research rely on an unrealistic assumption that all participants can share metadata such as class labels and the total number of categories. This assumption, which we term Metadata-Dependent Supervised Learning (MDSL), fails to reflect the diversity and autonomy of real-world participants. In contrast, we propose MAZEL: Metadata-Agnostic Zero-shot Learning, a framework that eliminates the need for shared metadata by leveraging CLIP-based zero-shot classification. MAZEL enables more realistic and flexible decentralized learning, where clients can dynamically join or leave without requiring predefined output heads. Our contributions are fourfold: (1) We formalize the distinction between MDSL and MAZEL; (2) we show that standard claims about performance degradation and slow convergence in MSDL-based decentralized learning may not hold under MAZEL; (3) we provide benchmarks using up to 8–16 diverse datasets to rigorously evaluate newly proposed decentralized learning methods under real metadata-agnostic cases; and (4) we propose two-stage and cosine gossip schedulers to optimize communication efficiency.

Primary Area: datasets and benchmarks

Submission Number: 3068