Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset

Abstract

One concern with the rise of large language models lies with their potential for significant harm, particularly from pretraining on biased, obscene, copyrighted, and private information. Emerging ethical approaches have attempted to filter pretraining material, but such approaches have been ad hoc and failed to take into account context. We offer an approach to filtering grounded in law, which has directly addressed the tradeoffs in filtering material. First, we gather and make available the Pile of Law, a ∼256GB (and growing) dataset of open-source English-language legal and administrative data, covering court opinions, contracts, administrative rules, and legislative records. Pretraining on the Pile of Law may potentially help with legal tasks that have the promise to improve access to justice. Second, we distill the legal norms that governments have developed to constrain the inclusion of toxic or private content into actionable lessons for researchers and discuss how our dataset reflects these norms. Third, we show how the Pile of Law offers researchers the opportunity to learn such filtering rules directly from the data, providing an exciting new research direction in model-based processing. Warning: this paper contains quotations that may be offensive or upsetting. * Equal contribution.Preprint. Under review.

Pre-training Transformers on Indian Legal Text

“…However, the legal text being very unique and requiring handling of issues as described in Section 1, there is significant scope of improvement over these models, pre-trained on the general domain. There have been efforts to pre-train transformers on the legal domain: (i) Chalkidis et al (2020) pre-trained BERT-base on EU and UK legislation and court documents from the US, European Court of Justice (ECJ) and European Court of Human Rights (ECtHR), and released the LegalBERT model; (ii) Zheng et al (2021) proposed CaseLaw-BERT, pre-trained on a corpus of US case law documents and contracts; (iii) Henderson et al (2022) prepared a huge corpus of US, Canada and EU documents (not just case law), called Pile of Law, and trained BERT-large on the same to yield the PoLBERT model; (iv) Xiao et al (2021) released Lawformer, a Longformer (Beltagy et al, 2020) based model pre-trained on Chinese legal text. The details of the pre-training datasets are available in Table 1 3 Dataset for pre-training LMs on…”

“…The latter two models are based on the same architecture as BERT-base. For the sake of fair comparison, we did not choose PoLBERT (Henderson et al, 2022) as a baseline since it is based on BERT-large, which is inherently more powerful.…”

“…Again, Zheng et al (2021) pre-trained BERT-base on only US case law documents, which was shown to improve performance over US legal datasets such as CaseHOLD. More recently, Henderson et al (2022) pre-trained BERT-large on 'Pile of Law' (PoL), a huge legal corpus of US and EU documents. 1 Finally, the Lawformer model (Xiao et al, 2021), pre-trained on Chinese legal text, has been designed keeping in mind that legal documents are usually much longer than documents in the general domain.…”