Few-shot and Zero-shot Approaches to Legal Text Classification: A Case Study in the Financial Sector

Sarkar, Rajdeep; Ojha, Atul Kr.; Megaro, Jay; Mariano, John; Herard, Vall; McCrae, John P.

doi:10.18653/v1/2021.nllp-1.10

Cited by 5 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, Katz et al [54] applied GPT-4 to the Uniform Bar Examination, and Bommarito et al [55] applied it to the Uniform CPA Examination developed by the American Institute of Certified Public Accountants. Sarkar et al [56] evaluated multiple techniques, including LLMs (BERT), in zero/few-shot classification of legal texts. GPT models have already been applied to analyse legal cases-for example, to: annotate sentences' roles in Board of Veterans' Appeals (BVA) cases, such as finding, evidence, legal rule, citation or reasoning [57]; predict Supreme Court Justice decisions [58]; determine how well a case passage explains a statutory term [59]; or generate interpretations of a term based on such passages [60,61].…”

Section: Related Workmentioning

confidence: 99%

Empirical legal analysis simplified: reducing complexity through automatic identification and evaluation of legally relevant factors

Gray,

Savelka,

Oliver

et al. 2024

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

This paper investigates the potential for reducing the complexity of AI and Law and empirical legal studies projects through a novel annotation methodology that relies on GPT Family Models to assist human annotators. Improving the speed, cost and quality of annotation could greatly benefit such projects. In modelling types of legal claims, researchers in the fields of empirical legal studies and AI and Law have long relied on manually annotating factors in case texts. To demonstrate our methodology, we employ cases and factors regarding whether a police officer has constitutional authority to detain a motorist on the basis of the officer’s suspicion that the motorist is trafficking drugs. Our results demonstrate how recent advances in text analytics can reduce the burden of identifying factors in large numbers of cases and improve machine learning models’ predictions of case outcomes. This article is part of the theme issue ‘A complexity science approach to law and governance’.

show abstract

Section: Related Workmentioning

confidence: 99%

Empirical legal analysis simplified: reducing complexity through automatic identification and evaluation of legally relevant factors

Gray,

Savelka,

Oliver

et al. 2024

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

show abstract

“…Similarly, Katz et al (2023) successfully applied GPT-4 to the Uniform Bar Examination, and Bommarito et al (2023) to the Uniform CPA Examination developed by the American Institute of Certified Public Accountants. Sarkar et al (2021) investigated the potential of various techniques, including LLMs (BERT), in zero/few-shot classification of legal texts. Savelka et al (2023) employed GPT-4 to evaluate the explanatory value of case sentences that refer to a statutory term of interest.…”

Section: Related Workmentioning

confidence: 99%

The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts

Savelka,

Ashley

2023

Front. Artif. Intell.

View full text Add to dashboard Cite

The emergence of ChatGPT has sensitized the general public, including the legal profession, to large language models' (LLMs) potential uses (e.g., document drafting, question answering, and summarization). Although recent studies have shown how well the technology performs in diverse semantic annotation tasks focused on legal texts, an influx of newer, more capable (GPT-4) or cost-effective (GPT-3.5-turbo) models requires another analysis. This paper addresses recent developments in the ability of LLMs to semantically annotate legal texts in zero-shot learning settings. Given the transition to mature generative AI systems, we examine the performance of GPT-4 and GPT-3.5-turbo(-16k), comparing it to the previous generation of GPT models, on three legal text annotation tasks involving diverse documents such as adjudicatory opinions, contractual clauses, or statutory provisions. We also compare the models' performance and cost to better understand the trade-offs. We found that the GPT-4 model clearly outperforms the GPT-3.5 models on two of the three tasks. The cost-effective GPT-3.5-turbo matches the performance of the 20× more expensive text-davinci-003 model. While one can annotate multiple data points within a single prompt, the performance degrades as the size of the batch increases. This work provides valuable information relevant for many practical applications (e.g., in contract review) and research projects (e.g., in empirical legal studies). Legal scholars and practicing lawyers alike can leverage these findings to guide their decisions in integrating LLMs in a wide range of workflows involving semantic annotation of legal texts.

show abstract

“…BART in ZS-setting. Zero-shot (ZS) classification in NLP has been used to classify text on which a model is not specifically trained (Sarkar et al, 2021;Yin et al, 2019a;Ye et al, 2020). Here, we use the pre-trained BART-Large MNLI 1 Publicly available from http://eur-lex.europa.eu/ 2 Publicly available from http://www.legislation.gov.uk 3 Cases from the European Court of Justice (ECJ), also available from EURLEX, cases from HUDOC, the repository of the European Court of Human Rights (ECHR) (http://hudoc.echr.coe.int/eng), cases from various courts across the USA, see https://case.law and US contracts from EDGAR, the database of US Securities and Exchange Commission (SECOM) (https://www.sec.gov/edgar.shtml).…”

Section: Modelsmentioning

confidence: 99%

Automatic Classification of Legal Violations in Cookie Banner Texts

Van Hofslot,

Akdag Salah,

Gatt

et al. 2022

Proceedings of the Natural Legal Language Processing Workshop 2022

View full text Add to dashboard Cite

Cookie banners are designed to request consent from website visitors for their personal data. Recent research suggest that a high percentage of cookie banners violate legal regulations as defined by the General Data Protection Regulation (GDPR) and the ePrivacy Directive. In this paper, we focus on language used in these cookie banners, and whether these violations can be automatically detected, or not. We make use of a small cookie banner dataset that is annotated by five experts for legal violations and test it with state of the art classification models, namely BERT, LEGAL-BERT, BART in a zero-shot setting and BERT with LIWC embeddings. Our results show that none of the models outperform the others in all classes, but in general, BERT and LEGAL-BERT provide the highest accuracy results (70%-97%). However, they are influenced by the small size and the unbalanced distributions in the dataset.

show abstract

Few-shot and Zero-shot Approaches to Legal Text Classification: A Case Study in the Financial Sector

Cited by 5 publications

References 13 publications

Empirical legal analysis simplified: reducing complexity through automatic identification and evaluation of legally relevant factors

Empirical legal analysis simplified: reducing complexity through automatic identification and evaluation of legally relevant factors

The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts

Automatic Classification of Legal Violations in Cookie Banner Texts

Contact Info

Product

Resources

About