Refactoring is a software maintenance activity that combats software deterioration while preserving system behavior. Previous empirical approaches have shown the main benefits of refactoring to improve design quality and support software evolution. Therefore, many approaches have been proposed to automate refactoring which range from low-level refactorings available in most of existing development IDEs, like Eclipse or IntelliJ IDEA, to sophisticated approaches and tools to remove code smells and anti-patterns. Yet, there is reluctance in industry to adopt the later ones when the number and/or complexity of the refactoring(s) to be performed is high. In other cases, there is a fear that automated refactoring changes, despite being syntactically correct, may be too artificial for human readers. To the best of our knowledge no previous work contrasted anti-pattern refactoring automatically performed or made by real developers.

To collect quantitative evidence that helps practitioners to make an informed decision whether to adopt or not automated refactoring in their projects. We performed an empirical study with practitioners considering two experiments. (1) We invited developers to participate in a survey to classify refactorings based on their origin (automatically or manually generated), and rate them based on their perceived-quality. The refactorings proposed were performed by RePOR, an automated refactoring approach based on partial order reduction techniques, and freelancers that we hired to perform refactoring on 5 anti-pattern types found in 10 open source systems. Freelancers worked on the same code units, and the resultant code changes were evaluated using well-known code metrics. (2) We asked a different group of 30 freelancers to complete code comprehension tasks on the code units previously refactored by either developers or RePOR, and without disclosing the origin of the refactorings in question. We measured developers performance using the NASA task load index to estimate their effort, time spent, and percentage of correct answers.

The obtained results show that the developers surveyed struggle to identify RePOR refactorings applied to 3 out of the 5 anti-patterns types in our oracle. Quality wise, developers did not favor manual refactorings more than automated ones, with one remarkable exception: when refactoring Blob anti-pattern. With respect to the comprehension tasks and the impact on code understandability, the results show no statistically significant difference between the refactorings performed by developers and the automated ones.

Based on these results, we conclude that in certain scenarios, automated refactoring can be as effective as manual refactorings in terms of perceived quality, but with the advantage of faster execution and more consistency among changes. Automated approaches need to include semantic context to produce changes that read more natural and consequently attract more practitioners. For example, learning name conventions and code lexicon to better name new code entities when refactoring.