RIGA at SemEval-2016 Task 8: Impact of Smatch Extensions and Character-Level Neural Translation on AMR Parsing Accuracy

Abstract

Two extensions to the AMR smatch scoring script are presented. The first extension combines the smatch scoring script with the C6.0 rule-based classifier to produce a human-readable report on the error patterns frequency observed in the scored AMR graphs. This first extension results in 4% gain over the state-of-art CAMR baseline parser by adding to it a manually crafted wrapper fixing the identified CAMR parser errors. The second extension combines a per-sentence smatch with an ensemble method for selecting the best AMR graph among the set of AMR graphs for the same sentence. This second modification automatically yields further 0.4% gain when applied to outputs of two nondeterministic AMR parsers: a CAMR+wrapper parser and a novel character-level neural translation AMR parser. For AMR parsing task the character-level neural translation attains surprising 7% gain over the carefully optimized word-level neural translation. Overall, we achieve smatch F1=62% on the SemEval-2016 official scoring set and F1=67% on the LDC2015E86 test set.

SemEval-2017 Task 9: Abstract Meaning Representation Parsing and Generation

“…The Oxford system appears to be quite different from last year's neural submission (Foland and Martin, 2016) but nevertheless is a strong competitor. Finally, the top-scoring system, that of UIT-DANGNT-CLNLP, got a 0.61 Smatch, while last year's top scoring systems (Barzdins and Gosko, 2016;Wang et al, 2016) scored a 0.62, practically the same score. This, despite the fact that the evaluation corpora were quite different.…”

Controlled Natural Language

“…Robust machine learning techniques are necessary instead to map the arbitrary input sentences to their meaning representation in terms of PropBank and FrameNet [7], or the emerging Abstract Meaning Representation, AMR [8], which is based on PropBank with named entity recognition and linking via DBpedia [9]. AMR parsing has reached 67% accuracy (the F 1 score) on opendomain texts, which is a level acceptable for automatic summarization [10].…”

Oxford at SemEval-2017 Task 9: Neural AMR Parsing with Pointer-Augmented Attention

“…Consequently most AMR parsers are pipelines that make extensive use of additional resources. Neural encoder-decoders have previously been proposed for AMR parsing, but reported accuracies are well below the state-of-the-art (Barzdins and Gosko, 2016), even with sophisticated pre-processing and categorization (Peng et al, 2017). The end-to-end neural approach contrasts with approaches based on a pipeline of multiple LSTMs (Foland Jr and Martin, 2016) or neural network classifiers inside a feature-and resource-rich parser (Damonte et al, 2017), which have performed competitively.…”