Utility-Based Data Transfers Scheduling Between Distributed Computing Facilities
Authors: Xin Wang (Illinois Institute of Technology), Wei Tang (Argonne National Laboratory), Rajkumar Kettimuttu (Argonne National Laboratory), Zhiling Lan (Illinois Institute of Technology)
Abstract: Today’s scientific applications increasingly involve large amounts of input/output data that must be moved among multiple computing facilities via wide-area networks (WANs). The bandwidth of WANs, however, is growing at a much smaller rate and thus becoming a bottleneck. Moreover, the network bandwidth has not been viewed as a limited resource, and thus coordinated allocation is lacking. Uncoordinated scheduling of competing data transfers over shared network links results in suboptimal system performance and poor user experiences. To address these problems, we propose a data transfer scheduler to coordinate data transfers between distributed computing facilities over WANs. The scheduler allocates resources to data transfer requests based on user-centric utility functions in order to achieve maximum overall user satisfaction. We conducted trace-based simulation and demonstrated that our data transfer scheduling algorithms can considerably improve data transfer performance as well as quantified user satisfaction compared with traditional first-come, first-serve or short-job-first approaches.
Two-page extended abstract: pdf