Kitchen-Sink Cluster Creation for Scientific Computing

Large-scale computations are a critical part of scientific discovery. The demand for such computations is generally met in two ways. Traditionally, the computations are performed on high-performance computing clusters where the hardware (compute nodes and networking) is tightly controlled (the super-computing paradigm). More recently, however, computations are performed on heterogeneous collections of consumer devices such as desktop computers or mobile phones on an ad hoc basis (the volunteer computing paradigm). In this paper, we aim to establish a middle ground between these two paradigms by providing high computing throughput in an environment where there is a mix of high-performance machines and lower-end workstations, as is common in a research laboratory setting. Specifically, we use the actor model of concurrent computing to create a heterogeneous cluster of computers based on availability (a kitchen-sink approach) and dynamically distribute embarrassingly parallel calculations from a scientific computing workflow across it. We demonstrate that the kitchen-sink cluster can be a compelling alternative to the super- and volunteer computing paradigms. The cluster effectively utilizes all resources available for a given computation even when resources become unavailable due to general hardware/software failure or local user demands during execution.