FedMDP: A Federated Learning Framework to Handle System and Model Heterogeneity in Resource-Constrained Environments

AHMED IMTEAJ, M. Hadi Amini

Abstract:

The advancement of technology, improvement of network
infrastructures, and wide availability of internet open up
the door of new opportunities to perform on-device inference. Realizing the potential of such advancement, Federated
Learning (FL) was invented that facilitates the formation of a
powerful model without exposing user data. While successful, it does not consider the combinational case, where the
selected FL agents independently craft their local model with
heterogeneous architecture and perform computational tasks
based on their available resources. In the original FL model,
all agents need to agree on a uniform model architecture and
are assigned a uniform computational task. However, in a
real-life resource-constrained FL setting, agents may not be
interested to share their local model architecture details due to
privacy and security concerns. Also, the heterogeneous local
model architectures cannot be aggregated together on the FL
server following the traditional approaches. Moving forward,
we may observe straggler agents due to resource-constrained
environments, such that any FL agent may find a task as computationally challenging that can prolong the model convergence. To address the above-mentioned challenges regarding
agent’s local model and resource heterogeneity, we propose
an FL framework, FedMDP that can effectively handle federated agents possessing nonidentical local model structure as
well as variant local resources using knowledge distillation
and dynamic local task allocation techniques. We tested our
framework on MNIST and CIFAR100 dataset and observed
significant improvement in accuracy in a highly heterogeneous environment. By considering 10 uniquely designed
model of the agents, we achieved 15% gain on average compared to the accuracy of the traditional learning methods and
observed a few percent lower accuracy compared to the case
if the agents’ local datasets were pooled and made available
for all the network agents.