The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts

Abstract

The emergence of ChatGPT has sensitized the general public, including the legal profession, to large language models' (LLMs) potential uses (e.g., document drafting, question answering, and summarization). Although recent studies have shown how well the technology performs in diverse semantic annotation tasks focused on legal texts, an influx of newer, more capable (GPT-4) or cost-effective (GPT-3.5-turbo) models requires another analysis. This paper addresses recent developments in the ability of LLMs to semantically annotate legal texts in zero-shot learning settings. Given the transition to mature generative AI systems, we examine the performance of GPT-4 and GPT-3.5-turbo(-16k), comparing it to the previous generation of GPT models, on three legal text annotation tasks involving diverse documents such as adjudicatory opinions, contractual clauses, or statutory provisions. We also compare the models' performance and cost to better understand the trade-offs. We found that the GPT-4 model clearly outperforms the GPT-3.5 models on two of the three tasks. The cost-effective GPT-3.5-turbo matches the performance of the 20× more expensive text-davinci-003 model. While one can annotate multiple data points within a single prompt, the performance degrades as the size of the batch increases. This work provides valuable information relevant for many practical applications (e.g., in contract review) and research projects (e.g., in empirical legal studies). Legal scholars and practicing lawyers alike can leverage these findings to guide their decisions in integrating LLMs in a wide range of workflows involving semantic annotation of legal texts.

Empirical legal analysis simplified: reducing complexity through automatic identification and evaluation of legally relevant factors

“…Sarkar et al [56] evaluated multiple techniques, including LLMs (BERT), in zero/few-shot classification of legal texts. GPT models have already been applied to analyse legal cases—for example, to: annotate sentences’ roles in Board of Veterans’ Appeals (BVA) cases, such as finding, evidence, legal rule, citation or reasoning [57]; predict Supreme Court Justice decisions [58]; determine how well a case passage explains a statutory term [59]; or generate interpretations of a term based on such passages [60,61]. Other studies by Blair-Stanek et al , Nguyen et al and Janatian et al were focused on the capabilities of the GPT models to conduct legal reasoning [62–64], to model US Supreme Court cases [58], to give legal information to laypeople [65], and to support online dispute resolution [66].…”

Predictive Modeling and Deep Phenotyping of Obstructive Sleep Apnea and Associated Comorbidities through Natural Language Processing and Large Language Models

“…With LLMs, embeddings are numerical representations of words, phrases, or sentences that capture contextual information and understand relationships within large segments of text. They have been employed in various tasks, such as text retrieval and ranking (e.g., Qadrud-Din et al (2020)), text classification (e.g.,Chae and Davidson (2023)), and sentiment analysis (e.g., Savelka and Ashley (2023)). In this project, our focus lies in embeddings extracted from LLMs.…”

Can GPT Alleviate the Burden of Annotation?

“…To develop the prompt, we follow [19], and provide the model with almost an exact copy of the annotation guidelines provided to annotators in [1] (cf. [20] where only excerpts are used). We call this "guideline-prompting".…”