JuriBERT: A Masked-Language Model Adaptation for French Legal Text

Abstract

Language models have proven to be very useful when adapted to specific domains. Nonetheless, little research has been done on the adaptation of domain-specific BERT models in the French language. In this paper, we focus on creating a language model adapted to French legal text with the goal of helping law professionals. We conclude that some specific tasks do not benefit from generic language models pre-trained on large amounts of data. We explore the use of smaller architectures in domain-specific sub-languages and their benefits for French legal text. We prove that domain-specific pre-trained models can perform better than their equivalent generalised ones in the legal domain. Finally, we release JuriBERT, a new set of BERT models adapted to the French legal domain.

Pretrained Language Models v. Court Ruling Predictions: A Case Study on a Small Dataset of French Court of Appeal Rulings

“…Our dataset is on another type of litigation (habitual residency of children) and we focus on the manual construction of our dataset, instead of automatically constructing it. Finally, Douka et al (2021) introduced JuriBERT, trained on Légifrance, an official web-site publishing all French law and evaluated it on topic classification tasks for documents from the Cour de Cassation (highest court in France).…”

“…Language Models As regards pretrained language models, we used FlauBERT and CamemBERT (Martin et al, 2020), two generic-purpose pretrained models for French, as well as JuriBERT (Douka et al, 2021), a language model trained only on data from the legal domain.…”

“…Thanks to this annotation procedure, lead by law experts, we compare the usefulness of various types of inputs for several types of classifiers, including bag-of-ngrams classifiers and BERT-based classifiers. In particular, we used 3 French pretrained language models: FlauBERT and Camem-BERT (Martin et al, 2020) that were trained on generic-domain data, and JuriBERT (Douka et al, 2021), that has been specifically trained on French legal texts.…”

Italian Legislative Text Classification for Gazzetta Ufficiale

“…In the legal domain, text classification has an established tradition, both in the monolingual (Šarić et al, 2014;Papaloukas et al, 2021) and in the multi-lingual setting (Steinberger et al, 2006(Steinberger et al, , 2012Chalkidis et al, 2019;Avram et al, 2021;Chalkidis et al, 2021). Moreover, the large availability of legal data, produced by national and supranational public institutions, set the stage for the development of domain-adapted models (Chalkidis et al, 2020;Douka et al, 2021;Masala et al, 2021;Licari and Comandè, 2022). As for Italian, a multi-label classification system for bills has been proposed by De Angelis et al (2022), based on Bi-GRU architecture using static word embeddings and employing a dataset of 28k legal document tagged with the TESEO thesaurus.…”

GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding

“…Pretrained language models (PLMs; Devlin et al 2019;Liu et al 2019;Raffel et al 2020) have seen broad adaptation across various domains such as biology , healthcare (Alsentzer et al, 2019), law (Chalkidis et al, 2020;Douka et al, 2021), software engineering (Tabassum et al, 2020), and social media (Röttger and Pierrehumbert, 2021;Guo et al, 2021). These models benefit from in-domain corpora (e.g., PubMed for the biomedical domain) to learn domain-specific terms and concepts.…”