A Spectral Line Data Processing Pipeline for SKA Using DALiuGE

With the construction of the Square Kilometre Array (SKA), the world's largest radio telescope, a new era in the detection of 21 cm neutral hydrogen (HI) spectral line is about to be ushered in, and the SKA spectral line sciences present tremendous opportunities and unprecedented data processing challenges. In this paper, we propose a fast processing pipeline for spectral line data based on the distribution execution framework Data Activated Liu Graph Engine (DALiuGE), and compare it with the Message Passing Interface (MPI) based method. The proposed pipeline is proved to be more stable and efficient on several comparison experiments. The scalability and execution efficiency performance of SKA spectral line data fast processing pipeline using DALiuGE for data volume changes have been verified, and it is believed that the DALiuGE based pipeline can be a solution for more scientific tasks facing massive data processing on SKA Regional Centre (SRC).