Online RL-based cloud autoscaling for scientific workflows: Evaluation of Q-Learning and SARSA

Q-Learning and SARSA are two well-known reinforcement learning (RL) algorithms that have shown promising results in several application domains. However, their approach to build solutions is quite different. For example, SARSA tends to be more conservative than Q-Learning while exploring the solution space. Motivated by such differences, in this paper, we conducted an evaluation of both algorithms in the context of online workflow autoscaling in pay-per-use Clouds, where the goal is to learn optimal virtual machine scaling policies to optimize metrics such as execution time and monetary costs. To do so, we based our experiments on a state-of-the-art scaling strategy with encouraging results in learning optimal scaling policies for reducing execution time and monetary cost. We conducted experiments on simulated environments with four widespread benchmark workflows and two types of virtual machines. Results show that SARSA outperforms Q-Learning in almost all cases. For two workflows SARSA obtains significant gains of up to 40.8% in the first 100 and 300 episodes respectively and losses less than 6% in all episodes observed. In one workflow SARSA achieves significant gains up to 13.9% and no significant losses were observed. There was only one workflow with no significant gains and one significant loss (16.2%) in 1 of 50 observations. In summary, we found multiple stages where SARSA achieves significant and remarkable gains, and the rest of the time both algorithms had a similar performance. In general terms, we can observe that SARSA performs better for learning scaling policies in the Cloud considering workflow applications commonly used by the community to benchmark Cloud workflow resource allocation techniques. These represent interesting results to further drive the design and selection of RL-based autoscaling strategies to schedule workflow executions in the Cloud.