Contract Discovery: Dataset and a Few-Shot Semantic Retrieval Challenge with Competitive Baselines

Borchmann, Łukasz; Wiśniewski, Dawid; Gretkowski, Andrzej; Kosmala, Izabela; Jurkiewicz, Dawid; Szałkiewicz, Łukasz; Pałka, Gabriela; Kaczmarek, Karol; Kaliska, Agnieszka; Graliński, Filip

doi:10.18653/v1/2020.findings-emnlp.380

Cited by 11 publications

(16 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, to alleviate the dependence on user's expertise and improve retrieval effectiveness, a significant amount of effort [12,22,34,40] has been devoted to the automatic classification of legal documents and queries. The extraction of key legal concepts can also facilitate the retrieval process [2,39].…”

Section: Legal Information Retrieval and Extractionmentioning

confidence: 99%

See 1 more Smart Citation

Cross-Domain Contract Element Extraction with a Bi-directional Feedback Clause-Element Relation Network

Wang¹,

Song

Zhang³

et al. 2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Contract element extraction (CEE) is the novel task of automatically identifying and extracting legally relevant elements such as contract dates, payments, and legislation references from contracts. Automatic methods for this task view it as a sequence labeling problem and dramatically reduce human labor. However, as contract genres and element types may vary widely, a significant challenge for this sequence labeling task is how to transfer knowledge from one domain to another, i.e., cross-domain CEE. Cross-domain CEE differs from cross-domain named entity recognition (NER) in two important ways. First, contract elements are far more fine-grained than named entities, which hinders the transfer of extractors. Second, the extraction zones for cross-domain contract element extraction (CEE) are much larger than for cross-domain NER. As a result, the contexts of elements from different domains can be more diverse.We propose a framework, the Bi-directional Feedback cLause-Element relaTion network (Bi-FLEET), for the cross-domain CEE task that addresses the above challenges. Bi-FLEET has three main components: (1) a context encoder, (2) a clause-element relation encoder, and (3) an inference layer. To incorporate invariant knowledge about element and clause types, a clause-element graph is constructed across domains and a hierarchical graph neural network is adopted in the clause-element relation encoder. To reduce the influence of context variations, a multi-task framework with a bidirectional feedback scheme is designed in the inference layer, conducting both clause classification and element extraction. The experimental results over both cross-domain NER and CEE tasks show that Bi-FLEET significantly outperforms state-of-the-art baselines.

show abstract

Section: Legal Information Retrieval and Extractionmentioning

confidence: 99%

“…fact") in the contract. CEE can bring useful insights into contracts and it can facilitate downstream applications, such as relevant clause retrieval or risk assessment [2].…”

Section: Introductionmentioning

confidence: 99%

Cross-Domain Contract Element Extraction with a Bi-directional Feedback Clause-Element Relation Network

Wang¹,

Song

Zhang³

et al. 2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Prior NLP work on contract language is extensive, including summarization (Manor and Li, 2019;Keymanesh et al, 2020) information extraction and understanding (Anish et al, 2019;Borchmann et al, 2020;Agarwal et al, 2021), as well as corpus studies looking at intrinsic properties of contracts (Curtotti and McCreath, 2011;Simonson et al, 2019) or providing new annotations over contract language (Funaki et al, 2020).…”

Section: Prior Workmentioning

confidence: 99%

“…We avoid neural network methods for this task for a few reasons. Results from Borchmann et al (2020) suggest that pre-trained models trained on more general language struggle on the idiosyncrasies of contract language-at least those pretrained on non-legal data. Further, a resourceintensive slot detection system is not scalable in the desired environment.…”

Section: Introductionmentioning

confidence: 99%

Supervised Identification of Participant Slots in Contracts

Simonson¹

2021

Proceedings of the Natural Legal Language Processing Workshop 2021

View full text Add to dashboard Cite

This paper presents a technique for the identification of participant slots in English language contracts. Taking inspiration from unsupervised slot extraction techniques, the system presented here uses a supervised approach to identify terms used to refer to a genrespecific slot in novel contracts. We evaluate the system in multiple feature configurations to demonstrate that the best performing system in both genres of contracts omits the exact mention form from consideration-even though such mention forms are often the name of the slot under consideration-and is instead based solely on the dependency label and parent; in other words, a more reliable quantification of a party's role in a contract is found in what they do rather than what they are named.

show abstract

“…Its first part, made up of 315 documents, was annotated by three annotators, except that only contexts with some similarity, pre-selected using methods based on semantic similarity (cf. [3]), were taken into account; this was to make the annotation faster and less-labor intensive. The second part, with 195 documents, was annotated entirely by hand.…”

Section: Nda Datasetmentioning

confidence: 99%

Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts

Stanisławek¹,

Graliński²,

Wróblewska³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The relevance of the Key Information Extraction (KIE) task is increasingly important in natural language processing problems. But there are still only a few well-defined problems that serve as benchmarks for solutions in this area. To bridge this gap, we introduce two new datasets (Kleister NDA and Kleister Charity). They involve a mix of scanned and born-digital long formal English-language documents. In these datasets, an NLP system is expected to find or infer various types of entities by employing both textual and structural layout features. The Kleister Charity dataset consists of 2,788 annual financial reports of charity organizations, with 61,643 unique pages and 21,612 entities to extract. The Kleister NDA dataset has 540 Non-disclosure Agreements, with 3,229 unique pages and 2,160 entities to extract. We provide several state-of-the-art baseline systems from the KIE domain (Flair, BERT, RoBERTa, LayoutLM, LAMBERT), which show that our datasets pose a strong challenge to existing models. The best model achieved an 81.77 % and an 83.57 % F1-score on respectively the Kleister NDA and the Kleister Charity datasets. We share the datasets to encourage progress on more in-depth and complex information extraction tasks.

show abstract

Contract Discovery: Dataset and a Few-Shot Semantic Retrieval Challenge with Competitive Baselines

Cited by 11 publications

References 29 publications

Cross-Domain Contract Element Extraction with a Bi-directional Feedback Clause-Element Relation Network

Cross-Domain Contract Element Extraction with a Bi-directional Feedback Clause-Element Relation Network

Supervised Identification of Participant Slots in Contracts

Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts

Contact Info

Product

Resources

About