Layer-wise partitioning and merging for efficient and scalable deep learning

Samson B. Akintoye, Liangxiu Han, Huw Lloyd, X. Zhang, Darren Dancey, Haoming Chen, Daoqiang Zhang

Abstract: Highlights•A novel layer-wise partition and merging model for efficient distributed computation.•The partitioning and merging mechanism minimises communication overhead in distributed training.•A forward pass and backward pass parallelisation method for solving locking problems.•Compared with state-of-the-art methods, validated on real data intensive cases.