A Comparative Analysis of Scheduling Mechanisms for Virtual Screening Workflow in a Shared Resource Environment

Traditional High-Throughput Computing (HTC) consists of running many loosely-coupled tasks that are independent but requires a large amount of computing power during significant period of time. However, recent emerging applications requiring millions or even billions of tasks to be processed within a relatively short period of time have expanded the traditional HTC into Many-Task Computing (MTC).In silico drug discovery offers an efficient alternative to reduce the cost of drug development and discovery process. For this purpose, virtual screening is used to select the most promising candidate drugs for in vitro testing from millions of chemical compounds. This process requires a substantial amount of computing resources and high-performance processing of docking simulations, which shows the typical characteristics of MTC applications. As the number of users performing this virtual screening process increases with limited available computing resources, it becomes crucial to devise an effective scheduling policy that can ensure a certain degree of fairness and user satisfaction.In this paper, we present a comparative analysis of scheduling mechanisms for the virtual screening workflow where multiple users in the system are sharing a common service infrastructure. To effectively support these multiple users, the underlying system should be able to consider fairness, user response time and overall system throughput. We have implemented two different scheduling algorithms which can address fairness and user response time respectively in a common middleware stack called HTCaaS which is a pilot-job based multi-level scheduling system running on top of a dedicated production-level cluster. Through our comparative analysis of two different scheduling mechanisms targeting different metrics on top of a single H/W and S/W system, we can give an insight to the research community on the design and implementation of a scheduling mechanism that can trade-off user fairness and overall system performance which is crucial to support challenging MTC applications.