Softbot Agent for Automatic GUI Testing: Computer Vision Layer

Abstract

In the context of testing, while there are automated testing tools available, many of them rely on interacting with the component at the logical level, perform tests at a basic level (testing isolated components), or interact at the graphical interface level but use techniques that are outdated compared to the latest advancements. The doctoral thesis of which this work is a part aims to build an agent that is independent of the development process of the system under test and can, therefore, perform the testing process without the biases arising from the authorship of the code to be tested or from the platform on which the program runs. This paper develops the computer vision layer that the agent will use to view the program’s graphical interface. To this end, a new dataset containing images from multiple platforms was created, and 13 different versions of object detection models were trained and compared in the task of identifying graphical interface components, resulting in a total of 325 distinct trained models. The results highlight the successful identification of key graphical components in applications and also identified future lines of research, both for improving the dataset and for training better models, focusing on those variants that achieved the best performance.