Achieving Optimal Block Pipelining in Organized Network Coded Gossip

Majid Khabbazian, Di Niu

Abstract: We use randomized network coding (RNC) with simple connection topology control to approach the theoretical limit on finish time of disseminating <inline-formula><tex-math>$k$</tex-math></inline-formula> blocks in a server cluster of <inline-formula><tex-math>$n$</tex-math></inline-formula> nodes. Unlike prior gossip literature which relies on completely random contact, we prove that with RNC, any receiver selection following a simple permutation rule can achieve a broadcast completion time of <inline-formula><tex-math>$k+n$</tex-math></inline-formula> and that a time-varying random ring topology achieves a completion time of <inline-formula><tex-math>$k+o(k)+O(\log \,n)$</tex-math> </inline-formula> , both with high probability. Since the theoretical limit on finish time is <inline-formula><tex-math> $k+\lceil \log _2 \,n\rceil$</tex-math></inline-formula> , our simple permutation algorithms achieve absolutely optimal (not only order-optimal) block pipelining for the <inline-formula><tex-math>$k$</tex-math></inline-formula> blocks. Our results hold for both one-to-all (broadcast) and all-to-all transfers. We demonstrate the usefulness of the proposed organized network coded gossip with an application to content distribution in cluster computing systems like MapReduce, and discuss practical block dividing strategies to hide the negative effect of computation overhead of network coding.