UnImplicit Shared Task Report: Detecting Clarification Requirements in Instructional Text

Roth, Michael; Anthonio, Talita

doi:10.18653/v1/2021.unimplicit-1.4

Cited by 4 publications

(4 citation statements)

References 4 publications

(6 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…6 For our submitted predictions on the test, which was evaluated by the organizers in terms of accuracy, measured as the ratio of correct predictions over all data instances (Roth and Anthonio, 2021), we achieved 66.3% accuracy for the mention-based 6 Our original submission had a bug, leading to low scores. We thus report results for our updated submission, without this bug, which is also reported in Roth and Anthonio (2021). system, which is higher than the logistic regression baseline provided by the organizers.…”

Section: Results and Analysismentioning

confidence: 99%

“…The shared task on implicit and underspecified language (Roth and Anthonio, 2021) 1 aims to provide a binary classification for revision requirements to make a prediction of whether sentences in instructional texts require revision to improve understandability. Since instructional texts must be clear enough so that readers and machines can actually achieve the goal described by the instructions, this task focuses on modeling implicit elements that make the sentence more precise and clear.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Mention-Based System for Revision Requirements Detection

Ruby¹,

Hardmeier²,

Stymne³

2021

Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language

View full text Add to dashboard Cite

Exploring aspects of sentential meaning that are implicit or underspecified in context is important for sentence understanding. In this paper, we propose a novel architecture based on mentions for revision requirements detection. The goal is to improve understandability, addressing some types of revisions, especially for the Replaced Pronoun type. We show that our mention-based system can predict replaced pronouns well on the mentionlevel. However, our combined sentence-level system does not improve on the sentence-level BERT baseline. We also present additional contrastive systems, and show results for each type of edit.

show abstract

Section: Results and Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Mention-Based System for Revision Requirements Detection

Ruby¹,

Hardmeier²,

Stymne³

2021

Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language

View full text Add to dashboard Cite

show abstract

“…Previous work, related to this involves a Shared Task (Roth and Anthonio, 2021) which was a binary classification task, in which systems had to predict whether a given sentence in context requires clarification or not. This shared task uses the same dataset that is the wikiHowToImprove dataset (Anthonio et al, 2020) but with some variations.…”

Section: Introductionmentioning

confidence: 99%

niksss at SemEval-2022 Task7:Transformers for Grading the Clarifications on Instructional Texts

Singh¹

2022

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)

View full text Add to dashboard Cite

This paper describes the 9th place system description for SemEval-2022 Task 7. The goal of this shared task was to develop computational models to predict how plausible a clarification made on an instructional text is. This shared task was divided into two Subtasks A and B. We attempted to solve these using various transformers-based architecture under different regime. We initially treated this as a text2text generation problem but comparing it with our recent approach we dropped it and treated this as a text-sequence classification and regression depending on the Subtask.

show abstract

“…Two sentences are considered equivalent (non-divergent) at the sentence level if the same overall information is conveyed, even if there are minor meaning differences. Finer-grained differences are not widely considered in the detection of semantic divergences, despite the fact that implicit information can be critical to the understanding of the sentence (Roth and Anthonio, 2021).…”

Section: Introductionmentioning

confidence: 99%

Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation

Wein¹,

Wang²,

Schneider³

2022

Preprint

View full text Add to dashboard Cite

Identifying semantically equivalent sentences is important for many cross-lingual and monolingual NLP tasks. Current approaches to semantic equivalence take a loose, sentencelevel approach to "equivalence," despite previous evidence that fine-grained differences and implicit content have an effect on human understanding (Roth and Anthonio, 2021) and system performance (Briakou and Carpuat, 2021). In this work, we introduce a novel, more sensitive method of characterizing semantic equivalence that leverages Abstract Meaning Representation graph structures. We develop an approach, which can be used with either gold or automatic AMR annotations, and demonstrate that our solution is in fact finer-grained than existing corpus filtering methods and more accurate at predicting strictly equivalent sentences than existing semantic similarity metrics. We suggest that our finer-grained measure of semantic equivalence could limit the workload in the task of human post-edited machine translation and in human evaluation of sentence similarity.

show abstract

UnImplicit Shared Task Report: Detecting Clarification Requirements in Instructional Text

Cited by 4 publications

References 4 publications

A Mention-Based System for Revision Requirements Detection

A Mention-Based System for Revision Requirements Detection

niksss at SemEval-2022 Task7:Transformers for Grading the Clarifications on Instructional Texts

Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation

Contact Info

Product

Resources

About