Threshold-Based Retrieval and Textual Entailment Detection on Legal Bar Exam Questions

Abstract

Getting an overview over the legal domain has become challenging, especially in a broad, international context. Legal question answering systems have the potential to alleviate this task by automatically retrieving relevant legal texts for a specific statement and checking whether the meaning of the statement can be inferred from the found documents. We investigate a combination of the BM25 scoring method of Elasticsearch with word embeddings trained on English translations of the German and Japanese civil law. For this, we define criteria which select a dynamic number of relevant documents according to threshold scores. Exploiting two deep learning classifiers and their respective prediction bias with a threshold-based answer inclusion criterion has shown to be beneficial for the textual entailment task, when compared to the baseline.

Iterative Self-Supervised Learning for Legal Similar Case Retrieval

“…Legal information retrieval can also be divided into two categories: sparse retrieval models and dense retrieval models. In the COLIEE 2019 competition, Wehnert [27] integrated BM25 scores with word centroid distances from word embeddings. This fusion, followed by applying a similarity threshold to varying document retrieval numbers per query, resulted in a refined set of analogous cases.…”

A Survey on Legal Question Answering Systems

“…Non-factoid questions are those whose answer is not directly accessible in the target document, which demands some inference and perhaps extra processing to obtain an answer. Wehnert et al have tried a new application of the popular BM25 approach (Wehnert et al, 2019), and the latest to date, (Verma et al, 2020) has had a focus on relevant subsection retrieval to answer the questions of legal nature. The story so far can be summarized in Table 5.…”