Large-Scale QA-SRL Parsing

Abstract

We present a new large-scale corpus of Question-Answer driven Semantic Role Labeling (QA-SRL) annotations, and the first high-quality QA-SRL parser. Our corpus, QA-SRL Bank 2.0, consists of over 250,000 question-answer pairs for over 64,000 sentences across 3 domains and was gathered with a new crowd-sourcing scheme that we show has high precision and good recall at modest cost. We also present neural models for two QA-SRL subtasks: detecting argument spans for a predicate and generating questions to label the semantic relationship. The best models achieve question accuracy of 82.6% and span-level accuracy of 77.6% (under human evaluation) on the full pipelined QA-SRL prediction task. They can also, as we show, be used to gather additional annotations at low cost.

Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling

“…In this more realistic setting, where the predicate must be predicted, our model achieves state-of-the-art performance on PropBank. It also reinforces the strong performance of similar span embedding methods for coreference , suggesting that this style of models could be used for other span-span relation tasks, such as syntactic parsing (Stern et al, 2017), relation extraction (Miwa and Bansal, 2016), and QA-SRL (FitzGerald et al, 2018). We consider the space of possible predicates to be all the tokens in the input sentence, and the space of arguments to be all continuous spans.Our model decides what relation exists between each predicate-argument pair (including no relation).…”

Break It Down: A Question Understanding Benchmark

“…Recent work has focused on cheaply eliciting quality annotations from nonexperts through crowdsourcing (He et al, 2016;Iyer et al, 2017;. FitzGerald et al (2018) facilitated non-expert annotation by introducing a formalism expressed in natural language for semantic-role-labeling. This mirrors QDMR, as both are expressed in natural language.…”

On the Limits of Learning to Actively Learn Semantic Representations

“…We evaluate the potential of the oracle policy on QA-SRL Bank 2.0 (FitzGerald et al, 2018). We use the training set of the science domain as D, randomly split it into S lab , S unlab , and S eval .…”

“…We compare the results of a base oracle policy (BASEORACLE) corresponding to the best policy we were able to obtain using the architecture from FitzGerald et al (2018) to the following baselines:…”

“…We set the number of candidate sets to K = 5, and the size of each set to L = 1, thus the size of the final labeled set is 510 examples. We train the publicly available span detection model released by FitzGerald et al (2018), which consumes as input a sentence x 1 , . .…”