Measuring Alignment Bias in Neural Seq2seq Semantic Parsers

Abstract

Prior to deep learning the semantic parsing community has been interested in understanding and modeling the range of possible word alignments between natural language sentences and their corresponding meaning representations. Sequence-to-sequence models changed the research landscape suggesting that we no longer need to worry about alignments since they can be learned automatically by means of an attention mechanism. More recently, researchers have started to question such premise. In this work we investigate whether seq2seq models can handle both simple and complex alignments. To answer this question we augment the popular GEO semantic parsing dataset with alignment annotations and create GEO-ALIGNED. We then study the performance of standard seq2seq models on the examples that can be aligned monotonically versus examples that require more complex alignments. Our empirical study shows that performance is significantly better over monotonic alignments. 1

Translate First Reorder Later: Leveraging Monotonicity in Semantic Parsing

“…It has also been conjectured that the lack of alignment information might hamper progress in semantic parsing (Zhang et al, 2019). As a result, the field has seen some annotation efforts in this regard (Shi et al, 2020;Herzig and Berant, 2021;Locatelli and Quattoni, 2022).…”

“…GEOALIGNED (Locatelli and Quattoni, 2022) augments the popular GEO semantic parsing benchmark (Zelle and Mooney, 1996) with token alignment annotations. The dataset contains questions about US geography and corresponding meaning representation using the FunQL formalism (Kate et al, 2005).…”

“…Locatelli and Quattoni (2022) showed that annotators are consistent in the way they pick head-tokens, and reported high inter-annotator agreement scores on GEOALIGNED.…”

“…Alignments can be modeled either implicitly through latent variables (Wang et al, 2021), or explicitly by leveraging gold alignment annotations (Shi et al, 2020;Liu et al, 2021a). We take the second direction and exploit a recently released multilingual dataset for semantic parsing annotated with word alignments: GEOALIGNED (Locatelli and Quattoni, 2022), which augments the popular GEO benchmark (Zelle and Mooney, 1996).…”