MDLoader: A Hybrid Model-driven Data Loader for Distributed Deep Neural Networks Training

Jonghyun Bae, Jong Youl Choi, Massimiliano Lupo Pasini, Kshitij Mehta, Khaled Z. Ibrahim

Abstract: In this work, we propose MD Loader, a hybrid in-memory data loader for distributed deep neural networks. MDLoader introduces a model-driven performance estimator to automatically switch between one-sided and collective communication at runtime.