LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

Abstract

Laws and their interpretations, legal arguments and agreements are typically expressed in writing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) technologies can be a valuable tool to support legal practitioners in these endeavors. Their usefulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improvements across multiple tasks.

Pre-training Transformers on Indian Legal Text

“…Thus, we only report the results for HierBERT with LSTM + Attn on top. Our model with InLegalBERT obtains higher performance over all results reported in Chalkidis et al (2022) for ECtHR-B -the macro-F1 obtained by our model is 75.88% compared to the best value of 74.7% reported in Chalkidis et al (2022).…”

“…These models have already obtained better results than those originally reported for two important benchmark tasks over Indian legal text -the LSI task over the ILSI dataset (Paul et al, 2022), and the CJPE task over the ILDC dataset . The InLegalBERT model also improves across two benchmarks over non-Indian legal text -the LSI task over the ECtHR-B dataset (Chalkidis et al, 2022) and the Semantic Segmentation task over the UKSS dataset . Given that the number of NLP studies on legal text is increasing rapidly in recent years, we hope that these LMs will benefit other researchers working on Legal NLP.…”

“…(ii) ECtHR-B: This is the dataset used in the ECtHR Task B from the LexGlue benchmark suite (Chalkidis et al, 2022). The ECtHR-B dataset consists of 11K examples (facts from cases argued in the European Court of Human Rights), split into train, dev and test.…”

NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly

“…Recent approaches explore modeling moral and ethical judgement of real-life anecdotes from Reddit (Emelin et al, 2021;Sap et al, 2019a;Lourie et al, 2021;Botzer et al, 2022), with DELPHI (Jiang et al, 2021a) unifying the moral judgement prediction on these related benchmarks. Related is another line of work modeling legal judgement on judicial corpora (Chalkidis et al, 2022).…”

Knowledge is Power: Understanding Causality Makes Legal judgment Prediction Models More Generalizable and Robust

“…Recently, the rapid development of large-scale pre-trained language models (PLMs) based on transformers significantly benefits this area (Cui et al, 2022). Some of the PLMs including BERT (Devlin et al, 2018) are further pre-trained on legal corpora, such as Legal-BERT (Chalkidis et al, 2020), exhibiting the SOTA performance on legal text processing benchmarks (e.g., LexGLUE) (Zheng et al, 2021;Chalkidis et al, 2022a). However, in the meantime, some severe problems of models are also discovered, including unfairness and discrimination (Chalkidis et al, 2022b).…”