Linguistically-Informed Self-Attention for Semantic Role Labeling

Abstract

Current state-of-the-art semantic role labeling (SRL) uses a deep neural network with no explicit linguistic features. However, prior work has shown that gold syntax trees can dramatically improve SRL decoding, suggesting the possibility of increased accuracy from explicit modeling of syntax. In this work, we present linguistically-informed self-attention (LISA): a neural network model that combines multi-head self-attention with multi-task learning across dependency parsing, part-ofspeech tagging, predicate detection and SRL.Unlike previous models which require significant pre-processing to prepare linguistic features, LISA can incorporate syntax using merely raw tokens as input, encoding the sequence only once to simultaneously perform parsing, predicate detection and role labeling for all predicates. Syntax is incorporated by training one attention head to attend to syntactic parents for each token. Moreover, if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model. In experiments on CoNLL-2005 SRL, LISA achieves new state-of-the-art performance for a model using predicted predicates and standard word embeddings, attaining 2.5 F1 absolute higher than the previous state-of-the-art on newswire and more than 3.5 F1 on outof-domain data, nearly 10% reduction in error. On ConLL-2012 English SRL we also show an improvement of more than 2.5 F1. LISA also out-performs the state-of-the-art with contextually-encoded (ELMo) word representations, by nearly 1.0 F1 on news and more than 2.0 F1 on out-of-domain text.

A Syntax-aware Multi-task Learning Framework for Chinese Semantic Role Labeling

“…He et al (2018b) propose a k-th order argument pruning algorithm based on systematic dependency trees. Strubell et al (2018) propose a self-attention based neural MTL model which incorporate dependency parsing as a auxiliary task for SRL. propose a MTL framework using hard parameter strategy to incorporate constituent parsing loss into semantic tasks, i.e.…”

Syntax-Enhanced Self-Attention-Based Semantic Role Labeling

“…By its nature, the self-attention-based model directly captures the relation between words in the sentence, which is convenient to incorporate syntactic dependency information. Strubell et al (2018) replace one attention head with pre-trained syntactic dependency information, which can be viewed as a hard way to inject syntax into the neural model. Enlightened by the machine translation model proposed by Shaw et al (2018), we introduce the Relation-Aware method to incorporate syntactic dependencies, which is a softer way to encode richer structural information.…”

Robust Semantic Parsing with Adversarial Learning for Domain Generalization

“…Several studies have tried to develop models with better generalization capacities (Yang et al, 2015), (FitzGerald et al, 2015). In recent works, PropBank SRL systems have evolved and span classifier approaches have been replaced by current state of the art sequence tagging models that use recurrent neural networks (He et al, 2017) and neural atten- tion (Tan et al, 2017;Strubell et al, 2018). However, these parsers still suffer performances drops of up to 12 points in F-measure on OOD with respect to ID.…”