Experiences Running Mixed Workloads on Cray Analytics Platforms

The ability to run both HPC and big data frameworks together on the same machine is a principal design goal for future Cray analytics platforms. HadoopTMprovides a reasonable solution for parallel processing of batch workloads using the YARN resource manager. SparkTMis a generalpurpose cluster-computing framework, which also provides parallel processing of batch workloads as well as in-memory data analytics capabilities; iterative, incremental algorithms; ad hoc queries; and stream processing. Spark can be run using YARN, MesosTMor its own standalone resource manager. The Cray Graph Engine (CGE) supports real-time analytics on the largest and most complex graph problems. CGE is a more traditional HPC application that runs under either Slurm or PBS. Traditionally, running workloads that require different resource managers requires static partitioning of the cluster. This can lead to underutilization of resources. In this paper, we describe our experiences running mixed workloads on our next generation Cray analytics platform (internally referred to as “Athena”) with dynamic resource partitioning. We discuss how we can run both HPC and big data workloads by leveraging different resource managers to interoperate with Mesos, a distributed cluster and resource manager, without having to statically partition the cluster. We also provide a sample workload to illustrate how Mesos is used to manage the multiple frameworks. Keywords-HPC, Big Data, Mesos, Marathon, Yarn, Slurm, Spark, Hadoop, CGE