Autonomic Orchestration of in-Situ and in-Transit Data Analytics For Simulation Studies.
Winter Simulation Conference(2023)
Abstract
Modern parallel/distributed simulations can produce large amounts of data. The historical approach of performing analyses at the end of the simulation is unlikely to cope with modern, extremely large-scale analytics jobs. Indeed, the I/O subsystem can quickly become the global bottleneck. Similarly, processing on-the-fly the data produced by simulations can significantly impair the performance in terms of computational capacity and network load. We present a methodology and reference architecture for constructing an autonomic control system to determine at runtime the best placement for data processing (on simulation nodes or a set of external nodes). This allows for a good tradeoff between the load on the simulation’s critical path and the data communication system. Our preliminary experimentation shows that autonomic orchestration is crucial to improve the global performance of a data analysis system, especially when the simulation node’s rate of data production varies during simulation.
MoreTranslated text
Key words
Data Processing,Large Amount Of Data,Computational Load,Critical Path,External Nodes,Reference Architecture,Digital Communication Systems,Amount Of Time,Dynamic Model,Data Generation,Completion Time,High-performance Computing,Load Data,Single Node,Number Of Simulations,Simulation Run,Static Configuration,Multiple Simulations,Serialized,Distributed Architecture,High-performance Computing Systems
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined