Generated tests in the context of maintenance tasks: a series of empirical studies.

Abstract

Maintenance tasks are often time-consuming and hard to manage. To cope with that, developers often use failing test cases to guide their maintenance efforts. Therefore, working with good test cases is essential to the success of maintenance. Automatically generated tests can save time and lead to higher code coverage. However, they often do not reflect realistic scenarios and include test smells. Moreover, it is not clear whether generated tests can be effective when guiding maintenance tasks, nor if developers fully accept them. In this work, we present a series of empirical studies that evaluate if automatically generated tests can support developers when maintaining code. First, we ran an empirical study with 20 real developers to compare how they perform maintenance tasks with automatically generated (Evosuite or Randoop) and manually-written tests. Our results showed that automatically generated tests can be a great help for identifying faults during maintenance. Developers were more accurate at identifying maintenance tasks when using Evosuite tests and equally effective to create bug fixes when using manually written, Evosuite, and Randoop. Then, we applied a survey with 82 developers to assess developers’ perceptions of the use of Randoop tests (that presented the worst performance in the first study) refactored by removing 3 kinds of test smells. Results of this investigation showed that developers preferred refactored Randoop tests to the original ones. Finally, a third empirical study with 24 developers focused on evaluating the impact of these refactored Randoop tests on maintenance performance. We found that the refactorings applied did not improve their performance in detecting the faults. On the other hand, developers were more effective in fixing the faults using refactored Randoop tests.