Early detection of manual refactoring faults.

Abstract

A recent study states that about 90% of all refactorings are done manually. Manual refactoring edits are error prone, as refactoring requires developers to coordinaterelated transformations and to understand the complex inter-relationship between affected files, variables, and methods. In this context, regression tests suites are often used as safety net for decreasing the chances of introducing behavior changes while refactoring. However, due to the high costs related to handling massive test suites, there is a need for optimizing testing execution. Test case prioritization techniques propose new test execution orders fostering early fault detection. However, existing general-purpose prioritization techniques are not specifically designed for detecting refactoring-related faults. In this work we propose a refactoringaware strategy - (RBA) (Refactoring-Based Approach) – for prioritizing regression test case execution. RBA reorders an existing test suite, according to a set of proposed Refactoring Fault Models (RFMs), which comprise impact locations of certain refactorings. Although being the most used refactoring validation strategy in practice, regression suites might be inadequate. Inadequate test suites may prevent developers from initiating or performing refactorings due to the risk of introducing bugs. To complement testing validation and help developers to review refactorings, we propose REFDISTILLER, a static analysis approach for detecting missing and extra edits that deviate from a standard refactoring and thus may affect a program’s behavior. Both strategies (RBA and RefDistiller) focus on Java/JUnit systems and on a set of the most common refactoring types. Our evaluation using a data set composed by hard-to-identify refactoring faults shows that RBA improves the position of the first fault-revealing test case in 71% of the suites, also providing a better grouping rate (in 73% of the cases) for test cases in the prioritized sequence, when compared to well-know general purpose techniques. Regarding REFDISTILLER, it detects 97% of all faults from our data set with seeded refactoring faults, of which 24% are not detected by generated test suites. Moreover, in a study with open source projects. REFDISTILLER detects 22.1 times more anomalies than testing, with 94% precision on overage. Those results show that (i) RBA can considerably improve prioritization during perfective evolution, both by anticipating fault detection as well as by helping to giving more information about the defects earlier, and (ii) REFDISTILLER effectively complements dynamic analysis by finding additional anomalies, while providing information that help fault debugging/fixing.