Transformer-Based Neural Network for Answer Selection in Question Answering

Abstract

Answer selection is a crucial subtask in the question answering (QA) system. Conventional avenues for this task mainly concentrate on developing linguistic tools that are limited in both performance and practicability. Answer selection approaches based on deep learning have been well investigated with the tremendous success of deep learning in natural language processing. However, the traditional neural networks employed in existing answer selection models, i.e., recursive neural network or convolutional neural network, typically suffer from obtaining the global text information due to their operating mechanisms. The recent Transformer neural network is considered to be good at extracting the global information by employing only self-attention mechanism. Thus, in this paper, we design a Transformer-based neural network for answer selection, where we deploy a bidirectional long short-term memory (BiLSTM) behind the Transformer to acquire both global information and sequential features in the question or answer sentence. Different from the original Transformer, our Transformer-based network focuses on sentence embedding rather than the seq2seq task. In addition, we employ a BiLSTM rather than utilizing the position encoding to incorporate sequential features as the universal Transformer does. Furthermore, we apply three aggregated strategies to generate sentence embeddings for question and answer, i.e., the weighted mean pooling, the max pooling, and the attentive pooling, leading to three corresponding Transformer-based models, i.e., QA-TF WP , QA-TF MP , and QA-TF AP , respectively. Finally, we evaluate our proposals on a popular QA dataset WikiQA. The experimental results demonstrate that our proposed Transformer-based answer selection models can produce a better performance compared with several competitive baselines. In detail, our best model outperforms the state-of-the-art baseline by up to 2.37%, 2.83%, and 3.79% in terms of MAP, MRR, and accuracy, respectively.

Conclusion-Supplement Answer Generation for Non-Factoid Questions

“…Such sequential generation of next statement tokens, however, weakens the original meaning of the first statement (question). Recently, several models based on the Transformer (Vaswani et al 2017), such as for passage ranking (Nogueira et al 2019;Liu, Duh, and Gao 2018) and answer selection (Shao et al 2019), have been proposed to evaluate question-answering systems. There are, however, few Transformer-based methods that generate non-factoid answers.…”

A Hierarchical Structured Multi-Head Attention Network for Multi-Turn Response Generation

“…Multi-head attention is the foundation of Transformer, and it can jointly attend to information from different representation subspaces at different locations, thus facilitating more effective learning of long-term dependencies. In the recent literature, we have already witnessed the success of Transform for many tasks, such as dense video captioning [37], question answering [38] and relation extraction [39], which inspires us to improve multi-turn response generation with the multi-head attention mechanism.…”

A Hybrid Framework for Problem Solving of Comparative Questions

“…Shao et al [30] first propose a collaborative learning mode which learns the distributed representations of question and answer by CNN and Bi-LSTM. Then, Shao et al [31] use Bi-LSTM to acquire both global information and sequential features in the question or answer sentence. The most similar work to ours is by Tan et al [32], but they address non-factoid questions.…”

“…A question can have multiple ground-truth answers and an answer can correspond to multiple questions or various paraphrases of the same question. '', which can be answered with the following two single facts (triples), ('' '', ''7 '', '' 31 '') and ('' '', '' '', '' ''), respectively. Our Objective: 1) Given a collection of questions, identify all comparative questions, 2) Extract comparative elements from the identified questions, 3) Given a question q containing the extracted comparative elements and its answer candidate set C q = {a 1 , a 2 , .…”