A deep learning-based automated framework for functional User Interface testing

Context: The use of automation tools in software testing helps keep pace with the timeline of the deliverables. Over time with the inclusion of continuous integration/continuous delivery (CI/CD) pipelines, automation tools are becoming less effective. The testing community is turning to AI to help keep the pace. Objective: We study the use of transformers to automate the process of test case generation directly from the User Interface (UI) element description instead of relying on the test specification document from which test cases are extracted manually. We also demonstrate the capability of the proposed approach in repairing flaky tests. Method: We employ object detection algorithms EfficientDet and DEtectionTRansformer for detecting the elements from an application UI automatically without requiring a tester to locate complex-scripted UI elements. We also use Tesseract to automatically identify the text present on the UI elements. We transform the generated UI element description to actual test designer-written test cases using text-generation transformers like GPT-2 and T5. The generated test cases are then translated into executable test scripts using a simple parser. We carry out our cases study on 30 e-commerce applications. Results: The percentage of correct executable test cases generated by the framework employing EfficientDet is 93.82% and employing DEtectionTRansformer is 98.08%. The framework eliminates an average of 96.05% flakiness across the applications selected for the study. Conclusion: It is concluded that the proposed approach can be used with current automation tools in the industry to enhance their capability in generating test cases and repairing the flaky tests.