MarVeLScaler: A Multi-View Learning based Auto-Scaling System for MapReduce

To promote cloud computing from current pay-per-request model to truly pay-per-use, tenants are crying for automatic tools to auto-estimate the amount of resources for MapReduce jobs. Such tools call for accurately quantifying the relationship among workload, resources and completion time. Various prediction models have been proposed. However, none of these models takes virtual machines' (VMs) performance variance during a job's execution into consideration, leading to underestimate the required resources and exceed the job's deadline. To address this problem, we propose a multi-view deep learning model to capture real-time performance variance and automatically scale out the cloud cluster whenever necessary. We implement MarVeLScaler, a prototype system including two useful modules, namely, Scale Estimator and Scale Controller. Scale Estimator preliminarily estimates the required cluster size for a MapReduce job with given a concrete workload and deadline. During the runtime, Scale Controller adjusts the scale of the cluster according to its real-time running status to guarantee the job finished on time. We evaluate the performance of MarVeLScaler based on Hadoop in Alibaba Cloud. Experiments show that MarVeLScaler can provide 98.4% accuracy of prediction in determining initial cluster size, and save 30.8% of expense while still guaranteeing similar performance compared with the state-of-the-art methods.