Avaliando a capacidade de LLMS na identificação de erros de compilação em linhas de produto de software.

Abstract

Compilation is an essential process in the development of software product lines, such as Linux. However, identifying compilation errors in Software Product Lines (SPLs) is not trivial, since traditional compilers are not variation aware. Previous approaches have been proposed that identify some of these compilation errors using advanced techniques that require programmers to use. This study evaluates the effectiveness of Large Language Models (LLMs), specifically ChatGPT 4 and Le Chat Mistral, in identifying compilation errors in SPLs. Initially, 50 products in C++, Java, and C languages were tested, and later 30 LPMs in C, covering 17 different types of compilation errors. The two LLMs were evaluated based on their ability to correctly recognize and diagnose the errors. ChatGPT was able to identify 82% and 95% of compilation errors in products and LPS, while Le Chat Mistral achieved 56% and 78%, respectively. The analysis revealed that although LLMs can identify a range of compilation errors, specific challenges remain, especially in LPS environments with high variability. The study suggests the need for continued refinements in LLM models to improve their accuracy and usefulness in complex software development scenarios.