"Aprimorando a verificação de conformidade em programas baseadas em contratos".

Abstract

Testing is commonly used to check conformance in contract-based programs, as verification by formal proofs is hard to scale and static analysis is, sometimes, limited for detecting general nonconformances. Traditional test cases, with manually-provided data, may be ineffective in detecting subtle nonconformances that arise only after several instantiations and modifications in objects under test. Those nonconformances may signalize more subtle bugs, hindering the benefits of using contract-based programs. Random-generated tests with automatic test data generation, on the other hand, is a promising approach when more substantial testing is demanded. In this work, we propose and evaluate an approach, JMLOK 2.0, for automatically detecting and categorizing nonconformances, in the context of Java Modeling Language (JML). Our approach aims to help the programmer in the process of nonconformances correction. The detection is backed by Randomly-Generated Tests (RGT) approach. And the categorization is backed by heuristics-based approach. We perform two evaluations. First, we perform an evaluation of our detection approach and our manual categorization process: we detected 84 nonconformances in over 29 KLOC and 9 K lines of JML contracts (that we will refer as KLJML henceforth); applying our manual classification system we got that most detected nonconformances were classified as postcondition errors; we also observed that a nonconformance is detected after 2.54 top-level test case calls, in average, and the number of internal calls within the faulty test case call is an average of 2.23, providing evidence for the need of a more complex generated test structure in nonconformance detection; furthermore, we compare our approach with JET, an existing test-based approach for detecting nonconformances in JML programs, using a subset of programs from first study (6 KLOC and 5 KLJML). JMLOK 2.0 detected 30 nonconformances with Java coverage of 78.44% and JML coverage of 67.67%, while JET detected 9 nonconformances by covering 47.97% of Java and 56.97% of JML. Second, we perform an evaluation of our automatic categorization approach: we compare automatic and manual categorization and got a matches value of 0.73 (considering the nonconformances from first evaluation) indicating similarity between automatic and manual approaches; furthermore, we compare our results with the categorization performed by voluntary JML experts and we also observed some similarity.