COLIEE 2020: Methods for Legal Document Retrieval and Entailment

Rabelo, Juliano; Kim, Miyoung; Goebel, Randy; Yoshioka, Masaharu; Kano, Yoshinobu; Satoh, Ken

doi:10.1007/978-3-030-79942-7_13

Cited by 34 publications

(15 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The best performing team in Task 1 in the 2020 edition, for example, achieved an f1-score of 0.6774. For more information on the previous task formulation and approaches, please see the COLIEE 2020 summary [16]. Most of the participating teams applied traditional IR techniques such as BM25, transformer based methods such as BERT, or a combination of both.…”

Section: Resultsmentioning

confidence: 99%

Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021

Rabelo

Goebel

Kim

et al. 2022

Rev Socionetwork Strat

Self Cite

View full text Add to dashboard Cite

We summarize the 8th Competition on Legal Information Extraction and Entailment. In this edition, the competition included five tasks on case law and statute law. The case law component includes an information retrieval Task (Task 1), and the confirmation of an entailment relation between an existing case and an unseen case (Task 2). The statute law component includes an information retrieval Task (Task 3), an entailment/question answering task based on retrieved civil code statutes (Task 4) and an entailment/question answering task without retrieved civil code statutes (Task 5). Participation was open to any group based on any approach. Eight different teams participated in the case law competition tasks, most of them in more than one task. We received results from six teams for Task 1 (16 runs) and 6 teams for Task 2 (17 runs). On the statute law task, there were eight different teams participating, most in more than one task. Six teams submitted a total of 18 runs for Task 3, 6 teams submitted a total of 18 runs for Task 4, and 4 teams submitted a total of 12 runs for Task 5. Here we summarize the approaches, our official evaluation, and analysis on our data and submission results.

show abstract

Section: Resultsmentioning

confidence: 99%

Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021

Rabelo

Goebel

Kim

et al. 2022

Rev Socionetwork Strat

Self Cite

View full text Add to dashboard Cite

show abstract

“…We extract all the paragraph tags (<p>) and the list tags (<ol>, <ul>, <dl>, <li>) to get relevant information about the articles. This is motivated by the team TRC3 in the previous year of COLIEE [10], where they used the content in Japanese itself. However, we translate the fetched content to English using the google-trans-new package 8 .…”

Section: Methodsmentioning

confidence: 99%

“…As Beel et al [1] comment, the TF-IDF vectorization scheme is the most widely used approach for content-based filtering for recommender systems and related text mining domains. In the COLIEE competition multiple teams in the previous years used TF-IDF vectors with or without other representation methods to retrieve the relevant articles given a query [8,10]. In legal information retrieval, TF-IDF only is still a valuable baseline model because its results are easy to interpret for domain experts.…”

Section: Retrieval With Tf-idfmentioning

confidence: 99%

“…For us, it is particularly interesting that there are recent approaches for fine-tuning a language model specifically to obtain meaningful sentence embeddings [2,11]. In the previous COLIEE edition, the cyber team achieved the best performance among all teams using the universal sentence encoder, TF-IDF and a support vector machine for the case law retrieval task [10]. Hence, we assume that TF-IDF combined with sentence embeddings could also work well on the statute law retrieval task.…”

Section: Contextualmentioning

confidence: 99%

See 1 more Smart Citation

Legal norm retrieval with variations of the bert model combined with TF-IDF vectorization

Wehnert

Sudhi

Dureja

et al. 2021

Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law

View full text Add to dashboard Cite

In this work, we examine variations of the BERT model on the statute law retrieval task of the COLIEE competition. This includes approaches to leverage BERT's contextual word embeddings, finetuning the model, combining it with TF-IDF vectorization, adding external knowledge to the statutes and data augmentation. Our ensemble of Sentence-BERT with two different TF-IDF representations and document enrichment exhibits the best performance on this task regarding the F2 score. This is followed by a fine-tuned LEGAL-BERT with TF-IDF and data augmentation and our third approach with the BERTScore. As a result, we show that there are significant differences between the chosen BERT approaches and discuss several design decisions in the context of statute law retrieval.

show abstract

“…Query by document (QBD) [38,37], is a widely-used practice across professional, domain-specific retrieval tasks [33,35] such as scientific literature retrieval [25,9], legal case law retrieval [2,3,30,34], and patent prior art retrieval [13,28]. In these tasks, the user's information need is based on a seed document of the same type as the documents in the collection.…”

Section: Introductionmentioning

confidence: 99%

Improving BERT-based Query-by-Document Retrieval with Multi-Task Optimization

Abolghasemi¹,

Verberne²,

Azzopardi³

2022

Preprint

View full text Add to dashboard Cite

Query-by-document (QBD) retrieval is an Information Retrieval task in which a seed document acts as the query and the goal is to retrieve related documents -it is particular common in professional search tasks. In this work we improve the retrieval effectiveness of the BERT re-ranker, proposing an extension to its fine-tuning step to better exploit the context of queries. To this end, we use an additional document-level representation learning objective besides the ranking objective when fine-tuning the BERT re-ranker. Our experiments on two QBD retrieval benchmarks show that the proposed multi-task optimization significantly improves the ranking effectiveness without changing the BERT re-ranker or using additional training samples. In future work, the generalizability of our approach to other retrieval tasks should be further investigated.

show abstract

COLIEE 2020: Methods for Legal Document Retrieval and Entailment

Cited by 34 publications

References 9 publications

Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021

Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021

Legal norm retrieval with variations of the bert model combined with TF-IDF vectorization

Improving BERT-based Query-by-Document Retrieval with Multi-Task Optimization

Contact Info

Product

Resources

About