Investigando o uso de testes para apoiar a resolução de problemas de programação.

Abstract

The introductory programming courses develop this skill through the resolution of several exercises related to the presented concepts, which also act as a way of evaluating the students’ performance. However, before producing the program, the student needs to understand what is required by the statement of the exercise. It is necessary to understand the specification as a first step for the effective resolution of programming problems. In this context, the objective of this master’s research is to investigate the effectiveness of a test-based strategy to better clarify the specification of the problem and, therefore, favor its correct resolution. We started the research by mapping the types of errors identified in programming exercises, with the aim of investigating whether the associated automated tests capture failures in understanding the problems. With the analysis of the data, it was possible to perceive that about 80% of the errors committed are related to difficulty in understanding or false assumptions raised based on the statements of the exercises. In this way, realizing that the automated tests seek to capture in detail certain scenarios specified in the statements of the exercises, in order to verify if the student understood them in their entirety, this preliminary study denotes the strategy based on tests as being promising. Therefore, we created the oracle to apply the approach and allow students to interact with the reference solution through tests, asking if the input of a problem corresponds with the output. In the main study, we performed a comparative analysis between the oracle and other problem solving strategies based on metrics associated with automated evaluation systems. The results show a significant improvement in the students’ performance, reducing the time to draw the correct solution, which was shorter in 65% of the cases; and also the number of submissions to the system used, which was lower in 68% of cases. With these effects added to the positive feedback of the participating students, we point out the strategy as being effective in solving programming problems.