Quão eficaz é uma ferramenta de automação de análise de dados baseada em LLM? um estudo de caso com o Data Analyst do ChatGPT.

Abstract

This paper investigates the effective application of Large Language Models (LLMs), specifically the OpenAI's Generative Pre-trained Transformer (ChatGPT), in data analysis. The relevance of this research emerges with the growing adoption of Artificial Intelligence (AI) tools in analytical processes, necessitating a meticulous evaluation of their capabilities and limitations to enhance decision-making and operational efficiency. Using the ChatGPT's Data Analyst as a case study, this work implements a structured experiment with 36 questions distributed across Descriptive, Diagnostic, Predictive, and Prescriptive analyses to measure its effectiveness. The results indicate na overall efficiency of 86,11%, with notable performance in descriptive and diagnostic analyses, while facing challenges in more complex categories, such as predictive and prescriptive. Despite technical limitations, such as data processing constraints and operational failures, the study underscores the significant potential of the Data Analyst in assisting data analysts, establishing an important milestone for future improvements and research in the practical application of LLMs in data analysis.