On Task Assignment and Scheduling for Distributed Job Execution

It is common for big data applications to run across multiple datacenters or machine clusters because the data inputs are distributed over different locations. This paper studies a job scheduling problem for distributed job execution in which the data inputs to jobs may be replicated across multiple locations so that each task of a job can be executed at any one of these locations. To schedule the jobs, we need to determine the processing locations for the tasks of each job and the execution order of the tasks at each location. We focus on the objective of minimizing the average job response time. We first design a task assignment algorithm to balance the task allocation among various locations. We then further develop integrated solutions that conduct task assignment and scheduling together. We experimentally evaluate our algorithms using real job traces. The results show that our algorithms can significantly reduce the job response times compared to a baseline that allocates each task to a fixed location for processing.