Coded MapReduce with Pre-set Data and Reduce Function Assignments

Yuhan Wang, Youlong Wu

Abstract: In this paper, we consider the general heterogeneous MapReduce system, where the file placement and Reduce function assignment are arbitrary but pre-set among all nodes (i.e., can not be designed by schemes). The storage and the computational capabilities for different nodes are not necessarily equal. We propose a universal CDC scheme, namely One-Shot Coded Transmission (OSCT), and establish the upper bound of the optimal communication load. The OSCT scheme encodes intermediate values into message blocks, each of which can be immediately and independently decoded by multiple intended nodes. We carefully design the bit-length of each message block to increase the multicasting gain. Furthermore, we provide a sufficient condition under which our scheme is optimal. To the best of our knowledge, this is the first work to investigate the general MapReduce problem with fixed data placement and Reduce function assignment.