ScriptPainter: Vision-based, On-device Test Script Generation for Mobile Systems

Yousung Choi, Ahreum Seo, Hyung-Sin Kim

Abstract: Dependability is an essential part of software engineering in the industry, which is the reason why most companies hire professional quality assurance (QA) engineers. Despite the presence of separate QA engineers, however, software developers are still required to perform basic QA processes by themselves before submitting their new software for peer review. When it comes to mobile system software, however, the basic QA process requires significant time and effort for writing different test scripts for various types of mobile devices, which degrades the productivity of developers. In this paper, we propose ScriptPainter, an easy-to-use automatic test script generator that enables a mobile phone to self-convert vision information on its screen into a test program; developers simply record a video while drawing a test scenario on a mobile touchscreen and ScriptPainter detects and tracks cursor locations in the video for generating a proper script. For accurate cursor detection and tracking with modest computation, ScriptPainter com-prises a set of techniques, such as frame sampling, deep neural network (DNN)-based detection, hough transform-based tracking with bounding box adaptation. ScriptPainter is evaluated on multiple Android smartphones, which shows that it analyzes a video 3.8x faster than pure DNN-based detection. Experiments with human developers show that ScriptPainter speeds up their job 8.3 times while providing even better accuracy compared to manual scripting.