Recursive Tree-Structured Self-Attention for Answer Sentence Selection

Abstract

Syntactic structure is an important component of natural language text. Recent topperforming models in Answer Sentence Selection (AS2) use self-attention and transfer learning, but not syntactic structure. Tree structures have shown strong performance in tasks with sentence pair input like semantic relatedness. We investigate whether tree structures can boost performance in AS2. We introduce the Tree Aggregation Transformer: a novel recursive, tree-structured self-attention model for AS2. The recursive nature of our model is able to represent all levels of syntactic parse trees with only one additional self-attention layer. Without transfer learning, we establish a new state of the art on the popular TrecQA and WikiQA benchmark datasets. Additionally, we evaluate our method on four Community Question Answering datasets, and find that tree-structured representations have limitations with noisy user-generated text. We conduct probing experiments to evaluate how our models leverage tree structures across datasets. Our findings show that the ability of treestructured models to successfully absorb syntactic information is strongly correlated with a higher performance in AS2.

Forming Trees with Treeformers

“…There is much prior work that induces, operates over, or otherwise uses a tree structure in neural net-work models (Socher et al, 2013a;Tai et al, 2015;Le and Zuidema, 2015;Dyer et al, 2016;Bradbury and Socher, 2017;Choi et al, 2017Choi et al, , 2018Drozdov et al, 2019;Ahmed et al, 2019;Wang et al, 2019;Mrini et al, 2021;Hu et al, 2021;Sartran et al, 2022). Such models are especially of interest due to the prevalence of trees in natural language.…”

Sentence-level Privacy for Document Embeddings

“…Language models have now become ubiquitous in NLP (Devlin et al, 2019;Liu et al, 2019b;Alsentzer et al, 2019), pushing the state of the art in a variety of tasks (Strubell et al, 2018;Liu et al, 2019a;Mrini et al, 2021). While language models capture meaning and various linguistic properties of text (Jawahar et al, 2019;Yenicelik et al, 2020), an individual's written text can include highly sensitive information.…”