Research and application of streaming Data transmission and processing architecture based on Pulsar

In order to solve the traditional message queue technology in the transmission of sensor data streams low transfer rate, high cost and lack of stream processing method, is proposed based on a stateless Pulsar message queue and the time sequence database IoTDB flow processing architecture, data transmission of hydrological data can provide efficient and reliable transmission, data preprocessing and early warning, provide the basis for subsequent data mining work. As a high performance message queue Kafka is often used in sensor data transmission scheme, Kafka is used to perform the same transmission task for comparative analysis. The experimental results show that Pulsar is able to perform the flow computation well, and has advantages in transmission speed and system resource utilization.