Learning Cognitive Features from Gaze Data for Sentiment and Sarcasm Classification using Convolutional Neural Network

Abstract

Cognitive NLP systems-i.e., NLP systems that make use of behavioral data -augment traditional text-based features with cognitive features extracted from eye-movement patterns, EEG signals, brain-imaging etc.. Such extraction of features is typically manual. We contend that manual extraction of features may not be the best way to tackle text subtleties that characteristically prevail in complex classification tasks like sentiment analysis and sarcasm detection, and that even the extraction and choice of features should be delegated to the learning system. We introduce a framework to automatically extract cognitive features from the eye-movement / gaze data of human readers reading the text and use them as features along with textual features for the tasks of sentiment polarity and sarcasm detection. Our proposed framework is based on Convolutional Neural Network (CNN). The CNN learns features from both gaze and text and uses them to classify the input text. We test our technique on published sentiment and sarcasm labeled datasets, enriched with gaze information, to show that using a combination of automatically learned text and gaze features often yields better classification performance over (i) CNN based systems that rely on text input alone and (ii) existing systems that rely on handcrafted gaze and textual features.

Deep learning for sentiment analysis: A survey

“…Mishra et al () utilized CNN to automatically extract cognitive features from the eye‐movement (or gaze) data to enrich information for sarcasm detection. Word embeddings are also used for irony recognition in English tweets (Van Hee, Lefever, & Hoste, ) and for controversial words identification in debates (Chen, Lin, & Ku, ).…”

Sequence Classification with Human Attention

“…They show that fixations not only have an impact in detecting sentiment, but also improve sarcasm detection. They train a convolutional neural network that learns features from both gaze and text and uses them to classify the input text (Mishra et al, 2017a). On a related note, Raudonis et al (2013) developed a emotion recognition system from visual stimulus (not text) and showed that features such as pupil size and motion speed are relevant to accurately detect emotions from eye-tracking data.…”

Assessing the quality of information on wikipedia: A deep‐learning approach

“…A pooling operation (max, sum, average) can be conducted on the feature map. The classification layer (dense layer) conducts classification that is based on the results from the pooling layer (Mishra, Dey, & Bhattacharyya, 2017). For the CNN-LSTMs, the CNN is used to extract features from the text representation.…”