A test fails and despite an investigation by a developer there is no fault found (NFF). Large software systems are often released with known failing and flaky tests. In this work, we quantify how often a test fails and does not find a fault. We conduct a case study on 9.9 million test runs of 10k tests across four releases of a large project at Ericsson.

For each test, we mine the rate of NFF test failures over total runs for each release, i.e. NFFRate. We compare the current level of test failure with the number of NFF failures during the stabilization period of the prior release, i.e. StableNFFRate. Using the binomial distribution, we are able to determine which tests exhibit a statistically larger number of failures relative to their expected StableNFFRate. These unstable tests need to be prioritized for re-run and potentially investigated to determine if there is a fault or if the test needs to be fixed or modified.

Our work has had an impact on Ericsson’s testing practices with testers using the NFFRate to determine which tests are the “flakiest" and need to be fixed or moved into an earlier, virtualized unit test stage. Testers also use our tool and technique to prioritize the statistically unstable tests failures for investigation and to examine longterm trends of test failures that may indicate a fault.