Test case prioritization: a case study in the evolution of a real system.

Abstract

The software maintenance can introduce new bugs in the system. Therefore, it is crucial to check the impacts caused on the rest of the code. One widely used methodology for validating program changes is regression testing. It consists of running the old tests on the new version of the system to check that there have been no changes in behavior. However, this process can be expensive and time-consuming. The Test Case Prioritization (TCP) is one of the strategies developed to help solve this problem. It rearranges the test suite based on a given criterion so that priority test cases are performed first. There are many TCP techniques available in the literature. And still, some developer’s teams still find their own ways of going through the regression costs. In this work, we conducted a case study on a large evolving system to evaluate and compare the effectiveness of six prioritization techniques, which use coverage information, with what happens in practice during software development. The developers of this system perform tests at two moments: before integrating the code, where it is required to run the entire suite, and during development, where they usually choose the tests to run to save time. Our objective in this work is to analyze the possible use of TCP techniques during the process of developing and integrating the code in this system. There are only a few industrial case studies involving real faults and evaluating TCP techniques that use coverage information in a system with a continuous integration environment. Our results showed that: 1) the modification-based techniques are the most effective in fault detection and reduction of the spread rate of failed tests, among the evaluated TCP techniques. 2) In the scenario where all the tests need to be performed, all the TCP techniques presented a better result than the original order. 3) Although TCP techniques had better results than the original order when all the tests need to be run, they were not more effective at fault detection than developer selection when time constraints are higher. They usually select the same number, or less, of failed tests using the same execution time. 4) The developer selection is maybe more effective because he almost always selects changed/added tests, which are the ones that usually fail, whereas the techniques do not prioritize these tests. Added or altered tests should always be considered when proposing a new prioritization technique.