Write a Blog >>
ESEC/FSE 2021
Thu 19 - Sat 28 August 2021 Clowdr Platform

Context: It has been argued that software engineering (SE) replications are useful for verifying the results of previous experiments. However, it has not yet been agreed how to check whether the results hold across replications. Besides, some authors suggest that replications that do not verify the results of previous experiments can be used to identify contextual variables causing the discrepancies.

Objective: Study how to assess the (dis)similarity of the results of SE replications when they are compared to verify the results of previous experiments and understand how to identify whether contextual variables are influencing results.

Method: We run simulations to learn how different ways of comparing replication results behave when verifying the results of previous experiments. We illustrate how to deal with context-induced changes. To do this, we analyze three groups of replications from our own research on test-driven development and testing techniques.

Results: The direct comparison of p-values and effect sizes does not appear to be suitable for verifying the results of previous experiments and examining the variables possibly affecting the results in software engineering. Analytical methods such as meta-analysis should be used to assess the similarity of software engineering replication results and identify discrepancies in results.

Conclusion: The results achieved in baseline experiments should no longer be regarded as a result that needs to be reproduced, but as a small piece of evidence within a larger picture that only emerges after assembling many small pieces to complete the puzzle.

Fri 27 Aug

Displayed time zone: Athens change

17:00 - 18:00
Human Aspects—Research Trends in Software EngineeringJournal First / Research Papers / Ideas, Visions and Reflections +12h
Chair(s): Ivica Crnkovic Chalmers University of Technology
17:00
10m
Paper
Comparing the Results of Replications in Software Engineering
Journal First
Adrian Santos Parrilla University of Oulu, Sira Vegas Universidad Politecnica de Madrid, Markku Oivo University of Oulu, Natalia Juristo Universidad Politecnica de Madrid
17:10
5m
Paper
Term Interrelations and Trends in Software Engineering
Ideas, Visions and Reflections
Janusan Baskararajah Ryerson University, Lei Zhang Ryerson University, Andriy Miranskyy Ryerson University
DOI Pre-print Media Attached
17:15
45m
Live Q&A
Q&A (Human Aspects—Research Trends in Software Engineering) & Mini-panel
Research Papers
P: Elvan Kula Delft University of Technology, P: Zoe Kotti Athens University of Economics and Business, P: Georgios Gousios Facebook & Delft University of Technology, P: Massimiliano Di Penta University of Sannio

Sat 28 Aug

Displayed time zone: Athens change

05:00 - 06:00
Human Aspects—Research Trends in Software EngineeringJournal First / Research Papers / Ideas, Visions and Reflections
Chair(s): Yuxia Zhang Beijing Institute of Technology
05:00
10m
Paper
Comparing the Results of Replications in Software Engineering
Journal First
Adrian Santos Parrilla University of Oulu, Sira Vegas Universidad Politecnica de Madrid, Markku Oivo University of Oulu, Natalia Juristo Universidad Politecnica de Madrid
05:10
5m
Paper
Term Interrelations and Trends in Software Engineering
Ideas, Visions and Reflections
Janusan Baskararajah Ryerson University, Lei Zhang Ryerson University, Andriy Miranskyy Ryerson University
DOI Pre-print Media Attached
05:15
45m
Live Q&A
Q&A (Human Aspects—Research Trends in Software Engineering) & Mini-panel
Research Papers
P: Elvan Kula Delft University of Technology, P: Zoe Kotti Athens University of Economics and Business, P: Georgios Gousios Facebook & Delft University of Technology, P: Massimiliano Di Penta University of Sannio