ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation

Abstract

An automated system that could assist a judge in predicting the outcome of a case would help expedite the judicial process. For such a system to be practically useful, predictions by the system should be explainable. To promote research in developing such a system, we introduce ILDC (Indian Legal Documents Corpus). ILDC is a large corpus of 35k Indian Supreme Court cases annotated with original court decisions. A portion of the corpus (a separate test set) is annotated with gold standard explanations by legal experts. Based on ILDC, we propose the task of Court Judgment Prediction and Explanation (CJPE). The task requires an automated system to predict an explainable outcome of a case. We experiment with a battery of baseline models for case predictions and propose a hierarchical occlusion based model for explainability. Our best prediction model has an accuracy of 78% versus 94% for human legal experts, pointing towards the complexity of the prediction task. The analysis of explanations by the proposed algorithm reveals a significant difference in the point of view of the algorithm and legal experts for explaining the judgments, pointing towards scope for future research.

Corpus for Automatic Structuring of Legal Documents

“…The project is hosted at https://legal-nlp-ekstep.github. io/Competitions/Rhetorical-Role/ (Jackson et al, 2003), Summarization (Moens et al, 1999;Saravanan et al, 2007), Case Prediction (Malik et al, 2021;Chalkidis et al, 2019;? ;S ¸ulea et al, 2017), Argument Mining (Wyner et al, 2010;Moens et al, 2007), Information Extraction and Retrieval (Tran et al, 2019;Grabmair et al, 2011;Tran et al, 2019), and Event Extraction (Lagos et al, 2010;Maxwell et al, 2009;Lagos et al, 2010).…”

“…Recently, efforts have been made to develop corpora that could aid various legal NLP tasks; for example, Malik et al (2021) have released a corpus of 35K Indian Supreme Court documents for the task of judgment prediction and explanation. Chalkidis et al (2019) have released 11,478 legal documents corresponding to the European Court of Human Rights (ECHR).…”

“…Towards this goal, we experimented with some baseline models. Since transformer-based models (Wolf et al, 2020) have shown state-of-the-art (SOTA) performance on most of the NLP tasks, including the tasks in legal NLP domain (Malik et al, 2021), we mainly experimented with them. Moreover, this is also true for.…”

“…Legal documents are quite long, running into tens (sometimes hundreds) of pages. Long documents make automatic processing challenging as information is spread throughout the document (Malik et al, 2021). Another challenge with legal documents is the use of different lexicons.…”

“…Though legal documents use natural language (e.g., English), many commonly occurring words/terms have different legal connotations. The use of different lexicons makes it challenging to adapt existing NLP models to legal texts (Malik et al, 2021). Moreover, in countries like India, legal documents are manually typed and are highly unstructured and noisy (e.g., spelling and grammatical mistakes).…”