Detectando fraquezas usando modelos de linguagem de grande porte: uma avaliação comparativa.

Abstract

The digital age demands secure software. Weaknesses in source code can have serious consequences, ranging from software failures to cyberattacks. Large Language Models (LLMs) such as ChatGPT 3.5, Gemini 1.0, Claude 3 Sonnet, and Mistral Large emerge as promising tools to aid in the detection of weaknesses in source code. This project evaluates the performance of LLMs in detecting 56 examples of weaknesses in source code. The results demonstrate that LLMs can be important tools for programmers. In our evaluation, LLMs detected 75% of the weaknesses of different types such as: SQL Injection, Cross-site Scripting, Out-of-bounds Write, and Null Pointer Dereference. Claude 3 Sonnet was the LLM with the best result.