Evaluating the effect of retrieval augmented generation in Mistral-7b-Instruct-v0.2’s clojure’s code review.

Abstract

Code review is one of the most important activities in software engineering, since it intends to guarantee code’s quality and reliability, but this process is done mostly manually, which can make it an onerous, time consuming and a failure susceptible task. The code review process is a strong candidate for automation in order to make it more efficient and less susceptible to failures due to its human component. In this work, we intend to explore the automation of the code review process by applying a LLM and an optimization technique in the context of Clojure’s code review, which is an emergent programming language. The LLM chosen was Mistral-7B-Instruct-v0.2 and the optimization technique was Retrieval Augmented Generation (RAG), both topics are discussed in the following sections of this work. Our results show that Mistral with and without the RAG optimization can review code like humans, but RAG didn’t improve the model’s review.