LLMs as tools for evaluating textual coherence: a comparative analysis.

Abstract

Textual coherence is fundamental for efective comprehension, determining the clarity, comprehensibility, and overall quality of content. Recent Large Language Models (LLMs), trained on extensive corpora, have demonstrated impressive capabilities in producing coherent and contextually relevant texts, enhancing their potential for textual analysis tasks. However, the ability of these models to perform coherence analysis on various input texts is still under investigation. In this study, we evaluate the performance of advanced language models in automatic textual coherence analysis. The models evaluated include GPT-4o, GPT-3.5, GPT-4, Claude Opus, Claude 3 Sonet, Claude 3 Haiku, Bard, LLaMA 2 13b, and LLaMA 2 7b. Our research investigates the ability of these models to evaluate textual coherence at diferent levels. First, we focus on local coherence, which refers to the logical and contextual consistency between adjacent sentences or small text segments. Our results indicate that GPT-4o, Claude Opus, and Gemini excel in this task, demonstrating superior performance in maintaining thematic continuity and Ćuency between consecutive sentences. Next, we explore global coherence, involving the logical and thematic consistency of entire texts. Here, the Claude Opus model proved to be the most efective, ensuring that the text maintains a consistent and logical Ćow from beginning to end. Finally, we examine the modelsŠ ability to identify incoherences, such as elements or segments that break the logical and thematic continuity. In this task, GPT-4o stood out, showing exceptional acuity in detecting and Ćagging incoherences. This aspect is crucial for applications where precision and clarity are needed, such as AI-assisted writing and text review. Our comparative analysis provides insights into the capabilities and limitations of current large language models in textual coherence analysis. Additionally, our Ąndings contribute to understanding how these models can be applied in various natural language processing contexts, promoting continuous advancements in this Ąeld.