SparkAIBench - A Benchmark to Generate AI Workloads on Spark.

With the rapid development of artificial intelligence and cloud computing technologies, more and more workloads embed AI algorithms are deploying on cloud systems. For lack of sufficient achievements on generating AI workloads in recent years, designing and developing an efficient benchmark for AI workloads will be significant helpful for optimization of job execution time and cluster resource utilization. SparkAIBench proposed in this paper is a user customized benchmark and is able to automatically generate a variety of AI workloads by transforming user requirements into JSON objects. Besides, cooperated with a DRL-based job scheduling optimizer, a real scenario is introduced in this paper to demonstrate how SparkAIBench works.