Scalable Run-time Data Indexing and Querying for Scientific Simulations

Scientific simulations running at scale on highend computing systems are generating tremendous amounts of raw data, which has to be carefully analyzed before scientists can derive insights from simulations and better understand the phenomena being modeled. Query-driven data analysis is an important technique used by scientists to gain insights from data, especially to capture intermittent transient information at simulation run-time. However, the increasing data volumes and associated I/O costs (latency and energy) are quickly making it infeasible to perform post-simulation query-driven data analysis in a cost-effective and timely manner. To address this challenge, in this paper, we present a scalable data indexing and querying framework using a data staging approach to support run-time query-driven data analysis for large scale scientific simulations. We achieve scalability, low overheads and minimal impact on the simulations by offloading computationally expensive indexing and querying to data staging servers. We also present an experimental evaluation, which demonstrates the performance and scalability of the framework.