On the Effectiveness of Pre-Trained Language Models for Legal Natural Language Processing: An Empirical Study

Abstract

We present the first comprehensive empirical evaluation of pre-trained language models (PLMs) for legal natural language processing (NLP) in order to examine their effectiveness in this domain.Our study covers eight representative and challenging legal datasets, ranging from 900 to 57K samples, across five NLP tasks: binary classification, multi-label classification, multiple choice question answering, summarization and information retrieval. We first run unsupervised, classical machine learning and/or non-PLM based deep learning methods on these datasets, and show that baseline systems' performance can be 4%∼35% lower than that of PLM-based methods. Next, we compare general-domain PLMs and those specifically pre-trained for the legal domain, and find that domain-specific PLMs demonstrate 1%∼5% higher performance than general-domain models, but only when the datasets are extremely close to the pretraining corpora. Finally, we evaluate six general-domain state-of-the-art systems, and show that they have limited generalizability to legal data, with performance gains from 0.1% to 1.2% over other PLM-based methods. Our experiments suggest that both general-domain and domain-specific PLM-based methods generally achieve better results than simpler methods on most tasks, with the exception of the retrieval task, where the best-performing baseline outperformed all PLM-based methods by at least 5%. Our findings can help legal NLP practitioners choose the appropriate methods for different tasks, and also shed light on potential future directions for legal NLP research.

Empirical legal analysis simplified: reducing complexity through automatic identification and evaluation of legally relevant factors

“…In the fields of NLP and AI and Law, researchers have applied automated techniques to classify texts of contracts (e.g. in terms of 41 categories involving general information, restrictive covenants, revenue risks [5], clause fairness [6], statutes by topics [6,7] and legal cases [8]).…”

“…The work on classifying legal cases has focused on classifying them by: argument organizational categories including fact, issue, rule/law/holding, analysis and conclusion/opinion/answer [9],judicial subtasks in connection with predicting judgments of civil law cases [10],whether the case overrules a previous one or by the type of procedural motion addressed [6],types of applicable legal claims [7],applicable civil code articles [11] and statutory elements [10],domain concepts [9],relevance to a query case [12],a winning or losing factual scenario for particular types of claims [7],factual features that strengthen or weaken a claim [13]. Items (i) and (vi) through (ix) are of special interest in empirical legal studies and, in particular, to the use of statistical methods such as ML algorithms, ‘to study legal doctrine through the use of fact-pattern analysis’ [14].…”

“…At the time of writing, costs can be found at this link: https://openai.com/pricing 6. We define the intersection as the shared factors between the model and gold-standard annotations.…”

Making a Computational Attorney

“…On one hand, the massive scale of complex text data enables or facilitates the (self-supervised) pre-training of L 3 M. On the other hand, the few-shot prompting (i.e., in-context learning) or zero-shot prompting capability of L 3 M for downstream tasks can greatly alleviate or even avoid the high labeling cost, while the flexibility of L 3 M to accommodate ambiguity and idiosyncrasies can help to meet the challenges of thoroughness and specialized knowledge. It is not surprising that with L 3 Ms such as LEGAL-BERT [6] and Lawformer [21], we are seeing new heights achieved in legal text classification and other tasks [25,15].…”

Evaluating the machine learning models based on natural language processing tasks

“…However, a significant leap in language model performance is achieved when these models are underpinned by neural networks. This integration of neural networks significantly broadens the spectrum of natural language processing (NLP) tasks that a language model can tackle [1]- [3]. A neural language model exhibits versatility in handling NLP tasks, spanning from straightforward to intricate challenges.…”