WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests

Xinyue Liu, Zihe Song, Weike Fang, Wei Yang, Weihang Wang

Keywords: Web End-to-End Testing, Flaky Tests, Test Automation, Browser UI Changes, Testing Reliability, Software Testing, Web Development

TL;DR: WEFix introduces an intelligent, automated approach to generate explicit waits in web end-to-end testing, ensuring a high success rate in fixing flaky tests while maintaining low runtime overhead.

Abstract:

Web end-to-end (e2e) testing evaluates the workflow of a web application from beginning to end. It simulates real-world user scenarios to ensure the application flows behave as expected. Web e2e testing plays an indispensable role in the development of modern web applications. However, web e2e tests are notorious for being flaky, i.e., the tests can produce inconsistent results, passing or failing unpredictably, despite no changes to the code under test.
One common type of flakiness in web e2e testing is caused by nondeterministic execution orders between the test code and the client-side code under test.
In particular, UI-based flakiness emerges as a notably prevalent and challenging issue to mitigate.
Such flaky tests are challenging to fix because the test code has limited knowledge about the client-side code execution.

In this paper, we propose WEFix, a technique that can automatically generate fix code for UI-based flakiness in web e2e testing. The core of our approach is to leverage browser UI changes to predict the client-side code execution and generate proper wait oracles. We evaluate the effectiveness and efficiency of WEFix against 122 web e2e flaky tests from seven popular real-world projects. Our results show that (1) UI-based flakiness is prevalent; (2) implicit waits introduce significant runtime overhead; (3) WEFix dramatically reduces the overhead (from 3.7$\times$ to 1.25$\times$) while still achieving a high correctness (98\%).

Track: Systems and Infrastructure for Web, Mobile, and WoT

Submission Guidelines Scope: Yes

Submission Guidelines Blind: Yes

Submission Guidelines Format: Yes

Submission Guidelines Limit: Yes

Submission Guidelines Authorship: Yes

Student Author: Yes

Submission Number: 1888