An Intelligent Labeling and Dataset Generating Method for Desktop Application Element Object Detection

Junda Zhu, Jun Ai, Tao Shi, Congran Zhang, Xinyue Wu

Abstract: As an important part of automatic interaction with personal computers, the precision of locating desktop application widgets via object detection models has a significant impact on the effectiveness of the subsequent interacting operations. In order to obtain massive data to further fine-tune models pretrained on datasets like MS COCO to adapt it to the downstream desktop application widgets object detection task, we introduce a data generating method and develop a GUI data generator to generate 12k GUI screenshots with ground truth GUI widgets coordinates, and build a dataset called DAWOD (Desktop Application Widgets Object Detection). We show that model with domain-specific fine-tuning on DAWOD will be better at identifying and locating desktop application widgets than itself without fine-tuning.