On the Role of Negative Precedent in Legal Outcome Prediction

Abstract

Every legal case sets a precedent by developing the law in one of the following two ways. It either expands its scope, in which case it sets positive precedent, or it narrows it, in which case it sets negative precedent. Legal outcome prediction, the prediction of positive outcome, is an increasingly popular task in AI. In contrast, we turn our focus to negative outcomes here, and introduce a new task of negative outcome prediction. We discover an asymmetry in existing models’ ability to predict positive and negative outcomes. Where the state-of-the-art outcome prediction model we used predicts positive outcomes at 75.06 F1, it predicts negative outcomes at only 10.09 F1, worse than a random baseline. To address this performance gap, we develop two new models inspired by the dynamics of a court process. Our first model significantly improves positive outcome prediction score to 77.15 F1 and our second model more than doubles the negative outcome prediction performance to 24.01 F1. Despite this improvement, shifting focus to negative outcomes reveals that there is still much room for improvement for outcome prediction models. https://github.com/valvoda/Negative-Precedent-in-Legal-Outcome-Prediction

Predicting the Amount of Compensation for Harm Awarded by Courts Using Machine-Learning Algorithms

“…Operating only on the texts of judgements should be perceived as an exception. When it comes to more recent literature, US Supreme Court decisions have been analysed (Katz et al, 2017), along with the European Court of Human Rights case law (Aletras et al, 2016;Medvedeva et al, 2020;Valvoda, Cotterell & Teufel, 2023) and judgements pronounced in France (Sulea et al, 2017), Germany (Waltl et al, 2017), the Philippines (Virtucio et al, 2018), UK (Strickson & La Iglesia, 2020), Turkey (Mumcuoğlu et al, 2021). Also noteworthy are the latest extensive literature reviews on the subject by Cui, Shen & Wen (2023) and by Medvedeva, Wieling & Vols (2023).…”

From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification

“…Previous works involving ECtHR corpus has dealt with COC (Aletras et al, 2016;Chalkidis et al, 2019;Valvoda et al, 2023;Santosh et al, 2022Santosh et al, , 2023Tyss et al, 2023;, argu-ment mining (Mochales and Moens, 2008;Habernal et al, 2023;Poudyal et al, 2019Poudyal et al, , 2020, event extraction (Filtz et al, 2020;Navas-Loro and Rodriguez-Doncel, 2022) and vulnerability type classification (Xu et al, 2023). In this work, we focus mainly on COC task using ECtHR corpus studying the sources of disagreement/HLV in COC rationale, to contribute to the field of explainable legal NLP.…”

CustodiAI: A System for Predicting Child Custody Outcomes

“…Despite the progress made in the field, Legal Judgment Prediction (LJP) faces several challenges, including the issue of AI model predictions' asymmetry in distinguishing positive and negative outcomes (Valvoda et al, 2023). This discrepancy is prevalent even with the emergence of models that simulate real-world court processes and human judge decision-making, indicating the complexity of the problem.…”