Magic Castle — Enabling Scalable HPC Training through Scalable Supporting Infrastructures

The potential HPC community grows ever wider as methodologies such as AI and big data analytics push the computational needs of more and more researchers into the HPC space. As a result, requirements for training are exploding as HPC adoption continues to gather pace. However, the number of topics that can be thoroughly addressed without providing access to actual HPC resources is very limited, even at the introductory level. In cases where access to production HPC resources is available, security concerns and the typical overhead of arranging for account provision and training reservations make the scalability of this approach challenging. Magic Castle aims to recreate the supercomputer user experience in public or private clouds. To define the virtual machines, volumes, and networks that are required in a cloud-provider agnostic way, it uses the open-source software Terraform and HashiCorp Language (HCL). These resources are then configured using the configuration management and deployment tool Puppet to replicate a virtual HPC infrastructure with a full scientific software stack, and including a feature-rich JupyterHub environment. The final resource is accessible both through a web browser and via SSH, making it trivially OS-agnostic for the trainees. Through the use of Magic Castle, we demonstrate that it is possible to dynamically provision virtual HPC system(s) in public or private cloud infrastructure easily, quickly, and cheaply. We also show that such infrastructures can support accelerators and fast interconnects, meaning that they can still be considered "true" HPC resources.