This extended abstract proposes the paper entitled “A Multi-Armed Bandit Approach for Test Case Prioritization in Continuous Integration Environments”, authored by Jackson A. Prado Lima and Silvia R. Vergilio for inclusion in the Journal First track of ESEC/FSE 2021. The paper was accepted for publication into the IEEE Transactions on Software Engineering (TSE) journal on May 4th, 2020, adhering to the period criterion established in the call for papers.

In this paper, we introduce a novel Test Case Prioritization (TCP) approach for the Continuous Integration (CI) context, namely COLEMAN (Combinatorial VOlatiLE Multi-Armed BANdit). The main motivation behind the use of MAB is to overcome some limitations of the existing approaches and to take into account some CI particularities. For instance, some approaches take time to execute, which is not suitable due to the CI environment restrictions. The great majority do not consider the dynamic environment of the software life-cycle in which new test cases can be added or removed (discontinued), fact known as test case volatility. We evaluate COLEMAN by carrying out experiments involving open-source real-world software systems, and different time budgets, reward functions, and MAB policies. We compared our approach against to a state-of-the-art approach widely used in the literature based on Reinforcement Learning (RL). A total of 11,210 experiments was performed, and the results were evaluated with both quality indicators and statistical significance tests, following the most recent best practice in the literature. The results show that our approach outperformed the RL approach in most of cases, regarding different metrics and independently of the time budget investigated.

The original journal paper can be found at https://doi.org/10.1109/TSE.2020.2992428, and our supplementary material containing assets such as datasets, results, technical report, and analysis can be found at https://doi.org/10.17605/OSF.IO/WMCBT.