Privacy-Preserving Multi-Granular Federated Neural Architecture Search - A General Framework

Zijie Pan, Li Hu, Weixuan Tang, Jin Li, Yi He, Zheli Liu

Abstract: Jointly learning from multiple datasets can help building versatile intelligent systems yet may give rise to serious concerns of data privacy and model selection. Specifically, on the one hand, these datasets can be distributed at various local clients, who may not be willing or do not ought to share data with each other. On the other hand, it is unrealistic to choose a model architecture that can well suit the disparate patterns and distributions carried by the various datasets in a priori. Whereas many works in federated learning [1] and neural architecture search [2] have been proposed to address one of the two concerns, very few have attempted the both. To close the gap, in this paper we deliver a framework, termed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Multi-Granular Federated Neural Architecture Search</i> (MGFNAS), to enable the automation of model architecture search in a federated and thus privacy-preserved setting. We argue that our MGFNAS framework is general in the sense that it does not impose any restriction on the search space or strategy, such that most existing neural architecture search techniques can be readily implemented in. The main idea of our framework is to search the optimal neural network architecture in two levels of granularity, enabling the neural-operator-based <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">micro-level</i> search and the cell-based <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">macro-level</i> search. The main challenge of implementing our framework lies in the fact that, due to the decentralized nature, the local architectures searched by multiple clients can differ drastically in order to fit their own datasets, while a general method to form the global model by aggregating the local architectures in both micro and macro levels is missing. To solve the issue, we propose a novel aggregation function, named Network Architecture Probabilistic Aggregation (NAPA). The key idea of our NAPA function is to treat the network architectures as graphs, of which the sub-graph structures being frequently appeared across multiple clients are modeled by probabilistic distributions. At each round, a global model is formed by sampling from those distributions in an exploration-exploitation fashion. Extensive experiments are carried out, and the results substantiate the viability and effectiveness of our proposed framework.