Pre-training Transformers on Indian Legal Text

Abstract

Natural Language Processing in the legal domain been benefited hugely by the emergence of Transformer-based Pre-trained Language Models (PLMs) pre-trained on legal text. There exist PLMs trained over European and US legal text, most notably Legal-BERT. However, with the rapidly increasing volume of NLP applications on Indian legal documents, and the distinguishing characteristics of Indian legal text, it has become necessary to pre-train LMs over Indian legal text as well. In this work, we introduce transformerbased PLMs pre-trained over a large corpus of Indian legal documents. We also apply these PLMs over several benchmark legal NLP tasks over both Indian legal text, as well as over legal text belonging to other domains (countries). The NLP tasks with which we experiment include Legal Statute Identification from facts, Semantic segmentation of court judgements, and Court Judgement Prediction. Our experiments demonstrate the utility of the Indiaspecific PLMs developed in this work.1 Although all these three models were named as Legal-BERT in the original research papers, we shall address them as LegalBERT (Chalkidis et al., 2020), CaseLawBERT (Zheng et al., 2021) and PoLBERT (Henderson et al., 2022) respectively, for sake of comprehension.

Nonet at SemEval-2023 Task 6: Methodologies for Legal Evaluation

“…We used the pre-trained transformers which are trained on general corpus like XLNet (Yang et al, 2019) and Roberta (Liu et al, 2019b) as well as trained on legal corpus LegalBERT (Chalkidis et al, 2020), InlegalBERT, IncaseLawBERT (Paul et al, 2022) and we also train the BERT large on Indian judgment cases. Since the transformers have restrictions that they can not accommodate more than 512 tokens, so we gave only the last 510 tokens (two special tokens are reserved for CLS and SEP) as (Malik et al, 2021) mentioned in the paper that in general most relevant information is present at the end of the documents.…”