Evaluating the impact of refactorings on the code comprehension of novices with eye tracking.

Abstract

Code comprehension is crucial for software maintenance and evolution, however, it can be hindered by structural problems in the code. To address the structural problems in the code and make it easier to read and understand, developers often use refactorings. Refactoring is a disciplined technique for restructuring the code that aims to improve its internal structure

preserving its behavior. However, the state-of-the-art on understanding the impact of refac- torings on code comprehension lacks conclusive results and quantitative/qualitative details

on how and why of possible correlations. While some studies found a divergent impact of

refactorings on code comprehension, another found that certain refactorings led to the intro- duction of more structural problems. Aiming to further investigate the impact of refactorings

on code comprehension, we conduct three controlled studies with eye tracking: the first one on the impact of clarified atoms of confusion with 32 novices in Python, the second one on the impact of Extract Method refactoring with 32 novices in Java, and the third one on the impact of #ifdef annotations with 64 majoritarily novices in the C language. Besides using multiple criteria such as time, the number of attempts, and opinions, we measured the visual effort of the subjects with eye tracking through fixation duration, fixations count, and regressions count. In our results, the clarified version of the code with an atom reduced the time to the extent of 38.6% and the number of answer attempts by 28%. In addition, we observed 47.3% fewer horizontal regressions count in the atom region, making its reading easier. The use of the Extract Method refactoring presented a significant reduction in the

time of two tasks, which varied from 70% to 78.8%. We observed an increase in the accu- racy of three tasks, which varied from 20% to 34.4%. The subjects solved these tasks with

the Extract Method going back visually in the code 74.4% to 84.6% less often compared to the Inline Method. In the context of #ifdef annotations, one of the refactorings adds one extra variable and two extra lines of code, which is only a small impact on the metric Lines of Code (LOC), but it presented reductions in the modified region by 46.9% in the time, 44.7% in the fixation duration, 48.4% in the fixation count, and 60.5% in the regressions count. These results raise educators’ awareness about certain refactorings and their potential to ease or hinder the code comprehension for novices in Python, Java, and C. Practitioners and language designers should be more careful when using or proposing refactorings that could possibly impair the novices’ abilities to understand the code. For researchers, these results show the potential of visual metrics to reveal an impact of refactorings that cannot be captured by static code metrics.