From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification

Abstract

In legal NLP, Case Outcome Classification (COC) must not only be accurate but also trustworthy and explainable. Existing work in explainable COC has been limited to annotations by a single expert. However, it is well-known that lawyers may disagree in their assessment of case facts. We hence collect a novel dataset RAVE: Rationale Variation in ECHR 1 , which is obtained from two experts in the domain of international human rights law, for whom we observe weak agreement. We study their disagreements and build a two-level task-independent taxonomy, supplemented with COC-specific subcategories. We quantitatively assess different taxonomy categories and find that disagreements mainly stem from underspecification of the legal context, which poses challenges given the typically limited granularity and noise in COC metadata. To our knowledge, this is the first work in the legal NLP that focuses on building a taxonomy over human label variation. We further assess the explainablility of state-of-the-art COC models on RAVE and observe limited agreement between models and experts. Overall, our case study reveals hitherto underappreciated complexities in creating benchmark datasets in legal NLP that revolve around identifying aspects of a case's facts supposedly relevant to its outcome.

On the Role of Negative Precedent in Legal Outcome Prediction

“…For example, if you want to know if and how a neural network's representations relate to a dependency parse of the input sentence, investigating the network's use of tokens in a sentence is a wise choice. In legal NLP, however, despite the recent rejuvenation of efforts in the field (Xu et al, 2023;Habernal et al, 2023), there is no generally accepted legal theory about how individual words or sentences affect the outcome of a case. We further elaborate on the related work in §9.…”

“…This makes our work different from prior research in two aspects. (1) Where the existing Legal AI work focuses on providing explanations by identifying relevant tokens (Xu et al, 2023), sentences (Malik et al, 2021) or paragraphs (Chalkidis et al, 2021), of the facts of the case, we find the entire relevant cases. (2) Instead of looking inside the input to the model, i.e., at the facts, we look at the training data of the model.…”

“…Despite the widespread recognition of the problem that unexplainable legal NLP systems pose (Hacker et al, 2020) and emerging work in the domain of case law summarization (Norkute et al, 2021), much remains to be done in order to build effective explainable legal NLP systems (Branting et al, 2019;Chalkidis et al, 2021). In particular, the existing work on explainable legal AI focuses on providing explanations by identifying relevant tokens (Xu et al, 2023), sentences (Malik et al, 2021) or paragraphs (Chalkidis et al, 2021), from the facts of the case. However, lawyers and judges typically reason at the level of a case.…”