Benchmarking elasticity of FaaS platforms as a foundation for objective-driven design of serverless applications

Application providers have to solve the trade-off between performance and deployment costs by selecting the "right" amount of provisioned computing resources for their application. The high value of changing this trade-off decision at runtime fueled a decade of combined efforts by industry and research to develop elastic applications. Despite these efforts, the development of elastic applications still demands significant time and expertise from application providers. To address this demand, FaaS platforms shift responsibilities associated with elasticity from the application developer to the cloud provider. While this shift is highly promising, FaaS platforms do not quantify elasticity; thus, application developers are unaware of how elastic FaaS platforms are. This lack of knowledge significantly impairs effective objective-driven design of serverless applications. In this paper, we present an experiment design and corresponding toolkit for quantifying elasticity and its associated trade-offs with latency, reliability, and execution costs. We present results for the evaluation of four popular FaaS platforms by AWS, Google, IBM, Microsoft, and show significant differences between the service offers. Based on our results, we assess the applicability of the individual FaaS platforms in three scenarios under different objectives: web serving, online data analysis, and offline batch processing.