Hydrological stream data pipeline framework based on IoTDB

With the increasing amount of hydrological data in Chuhe river basin, the traditional relational database has been unable to meet the needs of users, which not only makes it difficult to achieve low latency and high throughput in the real-time transmission of hydrological data, but also causes the phenomenon of long time or even system crash when querying large amount of annual water-level data. To solve this problem, this paper proposes a stream data pipeline framework based on timeseries databases IoTDB and Kafka, which can provide services for hydrological early warning and anomaly detection researchers. Based on the hydrological sensor data of Chuhe river, the processing scenarios of sensor stream data are set and compared with other NoSQL (HBase, MongoDB, RiakTS and Redis) in different scenarios. The performance and workload of different NoSQL in this data pipeline are tested. Finally, it is docked with Flink real-time stream data processing platform and compared with other data pipelines. The experimental results show that the stream data pipeline composed of IoTDB, Kafka and Flink is outstanding in data acquisition, transmission, incremental query and data analysis.