Investigation of test case prioritization for model-based testing.

Abstract

Software Testing is an expensive and laborious task among the Verification and Validation ones. Nevertheless testing is still the main way of assessing the quality of systems under development. In Model-Based Testing (MBT), test cases are generated automatically from system models, which provides time and cost reduction. On the other hand, frequently the size of the generated test suites leads to an unfeasible complete execution. Therefore, some approaches are used to deal with the costs involved in the test case execution, for example: Test Case Selection, Test Suite Reduction and the Test Case Prioritization (TCP). In order to improve resource consumption during system level testing, whereas not eliminating test cases, we explore TCP in our work. In this doctorate research we address the general problem of proposing a new order for the execution of system level test cases, specifically the ones generated through MBT approaches and considering that historical information is not available to guide this process. As a first step, we evaluate current techniques in this context, aiming at assessing their strengths and weaknesses. Based on these results, we propose two TCP techniques: HARP and CRISPy. The former uses indications (hints) provided by development team members regarding error prone portions of the system under test, as guidance for prioritization; the latter clusters test cases in order to add variability to the prioritization. HARP is based on the adaptive random strategy and explore test cases that cover steps related to the hints. To validate it, we perform two studies and from them we conclude that, with good hints, HARP can suggest effective prioritized test suites, provided that one avoid situations that the hints are not consensual in the development team. On the other hand, we propose CRISPy to complement HARP, i.e. to apply it when HARP is not indicated. To validate it we perform three studies and, even being a simple clustering technique, it already proposes effective prioritized test suites in comparison to current TCP techniques. Besides, depending on the kind of the test suites, choosing an adequate distance function could be a tuning step. We detail the implementation of all investigated and proposed techniques, and publish the artifacts related to the performed empirical studies in a companion site, enabling researchers to verify and reproduce our findings.