Avaliação de grandes modelos de linguagem quantizados na resolução de questões do ENEM.

Abstract

Although large language models (LLMs) represent a revolution in the way we interact with computers allowing the construction of complex questions and the ability to reason over a sequence of statements, their use is restricted due to the need for dedicated hardware for execution. In this study we evaluate the performance of LLMs based on the 7 and 13 billion LLaMA models, subjected to a quantization process and run on home hardware. The models considered were alpaca, koala, and vicuna. To evaluate the effectiveness of these models, we developed a database containing 1006 questions from the ENEM (National High School Exam). Our analysis revealed that the best performing model achieved an accuracy of approximately 40% for both the original texts of the Portuguese questions and their English translations. In addition, we evaluated the computational efficiency of the models by measuring the time required for execution. On average, the 7 and 13 billion LLMs took approximately 20 and 50 seconds, respectively, to process the queries on a machine equipped with an AMD Ryzen 5 3600x processor.