SANTOS, M. L. O.; http://lattes.cnpq.br/1426948852953123; SANTOS, Matheus Lisboa Oliveira dos.
Resumo:
Although large language models (LLMs) represent a revolution in the way we interact with computers
allowing the construction of complex questions and the ability to reason over a sequence of
statements, their use is restricted due to the need for dedicated hardware for execution. In this study
we evaluate the performance of LLMs based on the 7 and 13 billion LLaMA models, subjected to a
quantization process and run on home hardware. The models considered were alpaca, koala, and
vicuna. To evaluate the effectiveness of these models, we developed a database containing 1006
questions from the ENEM (National High School Exam). Our analysis revealed that the best
performing model achieved an accuracy of approximately 40% for both the original texts of the
Portuguese questions and their English translations. In addition, we evaluated the computational
efficiency of the models by measuring the time required for execution. On average, the 7 and 13
billion LLMs took approximately 20 and 50 seconds, respectively, to process the queries on a machine
equipped with an AMD Ryzen 5 3600x processor.