Semantic types for computational legal reasoning

Walker, Vern R.; Han, Ji Hae; Ni, Xiang; Yoseda, Kaneyasu

doi:10.1145/3086512.3086535

Cited by 16 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For each decision, the researchers extracted sentences addressing the factual issues. The sentences were then manually annotated with rhetorical roles they play in the respective decisions (Walker et al, 2017) The figure shows basic properties of the documents, i.e., sentences from adjudicatory decisions, for each of the semantic types. The lengths (min, mean, max) are measured in words.…”

Section: Bva Decisions Of Veterans Claimsmentioning

confidence: 99%

See 1 more Smart Citation

The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts

Savelka,

Ashley

2023

Front. Artif. Intell.

View full text Add to dashboard Cite

The emergence of ChatGPT has sensitized the general public, including the legal profession, to large language models' (LLMs) potential uses (e.g., document drafting, question answering, and summarization). Although recent studies have shown how well the technology performs in diverse semantic annotation tasks focused on legal texts, an influx of newer, more capable (GPT-4) or cost-effective (GPT-3.5-turbo) models requires another analysis. This paper addresses recent developments in the ability of LLMs to semantically annotate legal texts in zero-shot learning settings. Given the transition to mature generative AI systems, we examine the performance of GPT-4 and GPT-3.5-turbo(-16k), comparing it to the previous generation of GPT models, on three legal text annotation tasks involving diverse documents such as adjudicatory opinions, contractual clauses, or statutory provisions. We also compare the models' performance and cost to better understand the trade-offs. We found that the GPT-4 model clearly outperforms the GPT-3.5 models on two of the three tasks. The cost-effective GPT-3.5-turbo matches the performance of the 20× more expensive text-davinci-003 model. While one can annotate multiple data points within a single prompt, the performance degrades as the size of the batch increases. This work provides valuable information relevant for many practical applications (e.g., in contract review) and research projects (e.g., in empirical legal studies). Legal scholars and practicing lawyers alike can leverage these findings to guide their decisions in integrating LLMs in a wide range of workflows involving semantic annotation of legal texts.

show abstract

Section: Bva Decisions Of Veterans Claimsmentioning

confidence: 99%

“…The original PTSD data set from Walker et al (2017) contains 478 sentences that are simply annotated as Sentence. This is presumably a catch-all category reserved for sentences that do not fit any of the above definitions.…”

Section: Bva Decisions Of Veterans Claimsmentioning

confidence: 99%

The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts

Savelka,

Ashley

2023

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…Recent developments in closer collaboration between the argument mining community an the AI and Law community may change this in the future. Walker et al (2017) created a corpus of decisions adjudicating claims by US military veterans for disability compensation. By a careful analysis of the legal argumentation, they developed a representation of the legal argument which is designed for the extraction and representation of information about legal rules.…”

Section: Previous Work: Ai and Lawmentioning

confidence: 99%

Building a corpus of legal argumentation in Japanese judgement documents: towards structure-based summarisation

2019

View full text Add to dashboard Cite

We present an annotation scheme describing the argument structure of judgement documents, a central construct in Japanese law. To support the final goal of this work, namely summarisation aimed at the legal professions, we have designed blueprint models of summaries of various granularities, and our annotation model in turn is fitted around the information needed for the summaries. In this paper we report results of a manual annotation study, showing that the annotation is stable. The annotated corpus we created contains 89 documents (37,673 sentences; 2,528,604 characters). We also designed and implemented the first two stages of an algorithm for the automatic extraction of argument structure, and present evaluation results.

show abstract

Extracting Facts from Case Rulings Through Paragraph Segmentation of Judicial Decisions

Lou

Salaün

Westermann

et al. 2021

Natural Language Processing and Information Systems

View full text Add to dashboard Cite

Semantic types for computational legal reasoning

Cited by 16 publications

References 19 publications

The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts

The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts

Building a corpus of legal argumentation in Japanese judgement documents: towards structure-based summarisation

Extracting Facts from Case Rulings Through Paragraph Segmentation of Judicial Decisions

Contact Info

Product

Resources

About