Using Small Abstractions to Program Large Distributed Systems

Douglas Thain, Christopher Moretti, Hoang Bui, Li Yu, Nitesh V. Chawla, Patrick J. Flynn

Abstract: Distributed systems such as clusters, clouds and grids remain a difficult platform for executing large data intensive workloads. Even sophisticated users struggle to shape complex workloads into the simple 'assembly language' of file transfer and job submission. To address this, problem, we advocate the use of abstractions, which are simple frameworks for expressing large structured problems. In this talk, we will discuss three examples of abstractions developed at the University of Notre Dame for scientific applications. In each case, we have been able to scale up workloads one to wto orders of magnitude larger than we previously feasible. Through each example, we will address some persistent obstacles in the field of distributed computing.