Artificial intelligence for context-aware visual change detection in software test automation
CoRR(2024)
Abstract
Automated software testing is integral to the software development process,
streamlining workflows and ensuring product reliability. Visual testing within
this context, especially concerning user interface (UI) and user experience
(UX) validation, stands as one of crucial determinants of overall software
quality. Nevertheless, conventional methods like pixel-wise comparison and
region-based visual change detection fall short in capturing contextual
similarities, nuanced alterations, and understanding the spatial relationships
between UI elements. In this paper, we introduce a novel graph-based method for
visual change detection in software test automation. Leveraging a machine
learning model, our method accurately identifies UI controls from software
screenshots and constructs a graph representing contextual and spatial
relationships between the controls. This information is then used to find
correspondence between UI controls within screenshots of different versions of
a software. The resulting graph encapsulates the intricate layout of the UI and
underlying contextual relations, providing a holistic and context-aware model.
This model is finally used to detect and highlight visual regressions in the
UI. Comprehensive experiments on different datasets showed that our change
detector can accurately detect visual software changes in various simple and
complex test scenarios. Moreover, it outperformed pixel-wise comparison and
region-based baselines by a large margin in more complex testing scenarios.
This work not only contributes to the advancement of visual change detection
but also holds practical implications, offering a robust solution for
real-world software test automation challenges, enhancing reliability, and
ensuring the seamless evolution of software interfaces.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined