A Conceptual Design of Job Pre-processing Flow for Heterogeneous Batch Systems in Data Center

Processing a workflow (or a job) created by a user, who can be a researcher from a scientific laboratory or an analysis from a commercial organization, is the main functionality that a data center or a high-performance computing center is generally expected to provide. It can be accomplished with a single core processor and rather small amount of memory if the problem is adequately small while it may require thousands of nodes to solve a complicated problem and peta-bytes of storage for its output. Also specific applications on various platforms are required in general by users for resolving the problems appropriately. In this aspect, a data center should operate non-homogeneous systems for resource management, so-called batch system, in which it results in inefficient resource utilization due to stochastic behavior of user activity. Implementation of virtualization for resource management, e.g. Cloud Computing, is one of promising solutions recently arising, however, it results in the increase of complexity of the system itself as well as the system administration because it naturally implies the intervention of virtualization stack, e.g. hypervisor, between Operating System and applications for resource management. In this paper, we propose a new conceptual design to be implemented as a pre-scheduler capable to insert user submitted jobs dedicated to a specific batch system into available resources managed by other kind of batch systems. The proposed design features transparency in between clients and batch systems, accuracy in terms of monitoring and prediction on the available resources, and scalability for additional batch systems. We suggest the implementation example of the conceptual design based on the scenario established from our experience of operating a data center.