Learning Sentence Representation with Guidance of Human Attention

Abstract

Recently, much progress has been made in learning general-purpose sentence representations that can be used across domains. However, most of the existing models typically treat each word in a sentence equally. In contrast, extensive studies have proven that human read sentences efficiently by making a sequence of fixation and saccades. This motivates us to improve sentence representations by assigning different weights to the vectors of the component words, which can be treated as an attention mechanism on single sentences. To that end, we propose two novel attention models, in which the attention weights are derived using significant predictors of human reading time, i.e., Surprisal, POS tags and CCG supertags. The extensive experiments demonstrate that the proposed methods significantly improve upon the state-of-the-art sentence representation models.

Sequence Classification with Human Attention

“…On a related note, Raudonis et al (2013) developed a emotion recognition system from visual stimulus (not text) and showed that features such as pupil size and motion speed are relevant to accurately detect emotions from eye-tracking data. Wang et al (2017) use variables shown to correlate with human attention, e.g. surprisal, to guide the attention for sentence representations.…”

ACV-tree: A New Method for Sentence Similarity Modeling

“…Recently, owing to the success of word embeddings [Bengio et al, 2003;Mikolov et al, 2013], researchers have attempted to study sentence similarity modeling via sentence embeddings. This approach has become a successful paradigm in natural language processing (NLP) community [Kenter et al, 2016;Wang et al, 2017]; and particularly some studies have used the attention weight mechanism to further enhance the performance [Wang et al, 2017;Arora et al, 2017]. In this line of works, most previous studies focused on learning semantic information and modeling it as a continuous vector, while the syntactic information of sentences are not fully exploited.…”

“…So far, extensive studies have proven that word attributes, as represented by frequency, POS tag, length, Surprisal, etc., are all correlated with human reading time [Barrett et al, 2016]. Thereby, researchers have considered to assign words with different weights (known as attention weight mechanism), and there are many schemes to assign attention weights to words, such as smooth inverse frequency (SIF), term frequency-inverse document frequency (TF-IDF), Surprisal (SUR), POS tag (POS), CCG supertag (CCG) [Wang et al, 2017;Arora et al, 2017]. Tree kernel: Tree kernel is used to compute the similarity between structured trees.…”

Image Classification Based on Convolutional Denoising Sparse Autoencoder

“…They explore attention models over single sentences with guidance of human attention. In computer vision area, the core concept of attention models is to focus on the important parts of the input image, instead of giving all pixels the same weight [34]. Inspired by the theory of visual attention mechanism, we propose a sampling strategy about eye fixations based on human visual system.…”

“…Many attention models have been proposed in both natural language processing and computer vision. In [34], Wang et al have proven that human read sentences by making a sequence of fixations and saccades. They explore attention models over single sentences with guidance of human attention.…”