An Automated Framework for Distributed Deep Learning-A Tool Demo

Gharib Gharibi, Ravi Patel, Anissa Khan, Babak Poorebrahim Gilkalaye, Praneeth Vepakomma, Ramesh Raskar, Steve Penrod, Greg Storm, Riddhiman Das

Abstract: Split learning (SL) is a distributed deep-learning approach that enables individual data owners to train a shared model over their joint data without exchanging it with one another. SL has been the subject of much research in recent years, leading to the development of several versions for facilitating distributed learning. However, the majority of this work mainly focuses on optimizing the training process while largely ignoring the design and implementation of practical tool support. To fill this gap, we present our automated software framework for training deep neural networks from decentralized data based on our extended version of SL, termed Blind Learning. Specifically, we shed light on the underlying optimization algorithm, explain the design and implementation details of our framework, and present our preliminary evaluation results. We demonstrate that Blind Learning is 65% more computationally efficient than SL and can produce better performing models. Moreover, we show that running the same job in our framework is at least 4.5× faster than PySyft. Our goal is to spur the development of proper tool support for distributed deep learning.