Accelerating Time-to-Science by Streaming Detector Data Directly into Perlmutter Compute Nodes
CoRR(2024)
Abstract
Recent advancements in detector technology have significantly increased the
size and complexity of experimental data, and high-performance computing (HPC)
provides a path towards more efficient and timely data processing. However,
movement of large data sets from acquisition systems to HPC centers introduces
bottlenecks owing to storage I/O at both ends. This manuscript introduces a
streaming workflow designed for an high data rate electron detector that
streams data directly to compute node memory at the National Energy Research
Scientific Computing Center (NERSC), thereby avoiding storage I/O. The new
workflow deploys ZeroMQ-based services for data production, aggregation, and
distribution for on-the-fly processing, all coordinated through a distributed
key-value store. The system is integrated with the detector's science gateway
and utilizes the NERSC Superfacility API to initiate streaming jobs through a
web-based frontend. Our approach achieves up to a 14-fold increase in data
throughput and enhances predictability and reliability compared to a I/O-heavy
file-based transfer workflow. Our work highlights the transformative potential
of streaming workflows to expedite data analysis for time-sensitive
experiments.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined