Write a Blog >>
ESEC/FSE 2021
Thu 19 - Sat 28 August 2021 Clowdr Platform
Wed 25 Aug 2021 19:00 - 19:10 - Testing—Flaky Tests Chair(s): Wing Lam
Thu 26 Aug 2021 07:00 - 07:10 - Testing—Flaky Tests Chair(s): Wei Yang

Validation of Android apps via testing is difficult owing to the presence of flaky tests. Due to non-deterministic execution environments, a sequence of events (a test) may lead to success or failure in unpredictable ways. In this work, we present an approach and tool FlakeScanner for detecting flaky tests through exploration of event orders. Our key observation is that for a test in a mobile app, there is a testing framework thread which creates the test events, a main User-Interface (UI) thread processing these events, and there may be several other background threads running asynchronously.
For any event $e$ whose execution involves potential non-determinism, we localize the earliest (latest) event after (before) which $e$ must happen. We then efficiently explore the schedules between the upper/lower bound events while grouping events within a single statement, to find whether the test outcome is flaky. We also create a suite of subject programs called FlakyAppRepo (containing 33 widely-used Android projects) to study flaky tests in Android apps. Our experiments on the subject-suite FlakyAppRepo show FlakeScanner detected 45 out of 52 known flaky tests as well as 245 previously unknown flaky tests among 1444 tests.

Wed 25 Aug

Displayed time zone: Athens change

19:00 - 20:00
Testing—Flaky TestsIndustry Papers / Research Papers +12h
Chair(s): Wing Lam University of Illinois at Urbana-Champaign
19:00
10m
Paper
Flaky Test Detection in Android via Event Order Exploration
Research Papers
Zhen Dong National University of Singapore, Abhishek Tiwari National University of Singapore, Xiao Liang Yu National University of Singapore, Abhik Roychoudhury National University of Singapore
DOI Pre-print
19:10
10m
Paper
Quantifying No-Fault-Found Test Failures to Prioritize Inspection of Flaky Tests at Ericsson
Industry Papers
Maaz Hafeez Ur Rehman Concordia University, Peter C. Rigby Concordia University
DOI
19:20
10m
Paper
When Life Gives You Oranges: Detecting and Diagnosing Intermittent Job Failures at Mozilla
Industry Papers
Johannes Lampel CISPA; Saarland University, Sascha Just Microsoft, Sven Apel Saarland University, Andreas Zeller CISPA
DOI Pre-print
19:30
30m
Live Q&A
Q&A (Testing—Flaky Tests)
Research Papers

Thu 26 Aug

Displayed time zone: Athens change

07:00 - 08:00
Testing—Flaky TestsResearch Papers / Industry Papers
Chair(s): Wei Yang University of Texas at Dallas
07:00
10m
Paper
Flaky Test Detection in Android via Event Order Exploration
Research Papers
Zhen Dong National University of Singapore, Abhishek Tiwari National University of Singapore, Xiao Liang Yu National University of Singapore, Abhik Roychoudhury National University of Singapore
DOI Pre-print
07:10
10m
Paper
Quantifying No-Fault-Found Test Failures to Prioritize Inspection of Flaky Tests at Ericsson
Industry Papers
Maaz Hafeez Ur Rehman Concordia University, Peter C. Rigby Concordia University
DOI
07:20
10m
Paper
When Life Gives You Oranges: Detecting and Diagnosing Intermittent Job Failures at Mozilla
Industry Papers
Johannes Lampel CISPA; Saarland University, Sascha Just Microsoft, Sven Apel Saarland University, Andreas Zeller CISPA
DOI Pre-print
07:30
30m
Live Q&A
Q&A (Testing—Flaky Tests)
Research Papers