A Grey Literature Review on Data Stream Processing applications testing

The Data Stream Processing (DSP) approach focuses on real-time data processing by applying specific techniques for capturing and processing relevant data for on-the-fly results, i.e. without necessarily requiring prior storage. Like in any other software, testing plays a vital role in the quality assurance of DSP applications. However, testing such kind of software is not a simple task. In this context, some factors that make challenging testing are message temporality, parallelism, data volume, complex infrastructure, variability, and speed of messages. This work aims to map and synthesize industry knowledge and experience regarding DSP application testing. Specifically, we want to know about challenges, test purposes, test approaches, test data sources, and adopted tools. To achieve the objective, we performed a Grey Literature Review (e.g., blog posts, white papers, discussion lists, lecture themes at technical events, professional social networks, software repositories, and other web-published) on testing DSP applications. We searched the grey literature using Google’s regular search engine in addition to specific searches on technical software development content websites. The selected studies were analyzed using qualitative and quantitative techniques. Results are based on evidence from 154 selected sources. The challenges for testing DSP applications are the complexity of DSP applications, test infrastructure complexity, timing, and data acquisition issues. The main test objectives identified are functional suitability, performance efficiency, reliability, and maintainability. The main test approaches reported: Performance Testing, Regression Testing, Property-Based Testing, Chaos Testing, and Contract/Schema Testing. The strategies adopted by practitioners to obtain test data: Historical Data, Production Data Mirroring, Semi-Synthetic Data, and Synthetic Data. We also report 50 tools used in various testing activities, which are used for: automating infrastructure, generating test data, test utilities, dealing with timing issues, load generation, simulation, and others. Furthermore, we identified gaps and opportunities for future scientific work. This work selected and summarized content produced by practitioners regarding DSP application testing. We identified that knowledge, techniques, and tools intrinsic to the practice were not present in the formal literature, so this study helps reduce the gap between industry and academia on this topic. The document has delivered benefits to industry practitioners and academic researchers.