Cognitive Visual Commonsense Reasoning Using Dynamic Working Memory

Attention Mechanism based Cognition-level Scene Understanding

“…It focuses on encoding the relationship between sentences using sequence-to-sequence based encoding methods. These methods infer rationales by encoding the long dependency relationship between sentences (see, e.g., R2C [5] and TAB-VCR [10], DMVCR [13]). However, these models face difficulty with reasoning information lost based on long dependency structure, and it is hard for them to infer reason based on commonsense about the world.…”

“…-We theoretically analyze the inadequacy of previous work [13] and introduce a new effective model to reduce the sequential computation. In [13], the computational complexity of related signals from two positions grows with the distance between the positions, which results in difficulties in learning dependencies among positions for the sequential task. In the newly proposed multimodal feature fusion layer and commonsense encoder submodules, we designed a parallel attention structure to limit the number of operations.…”

Interpretable Visual Understanding with Cognitive Attention Network

“…R2C [5] is a representative example in this line of efforts in which attention based deep model is used for visual inferencing. More recently, a dynamic working memory based memory cells framework is proposed to provide prior knowledge for inference [14]. Our model more closely resembles this method with two distinctions: i) a parallel structure is explicitly designed to relax the dependence on the previous cells, alleviating the drawback of information lose of long dependency memory cell for long sequences, and ii) a newly proposed co-attention network rather than dynamic working memory cell to ease model training but also to enhance the capability of capturing relationship between sentences and semantic information from surrounding words.…”

“…We compare our method with several state-of-the-art visual scene understanding models based on the mean average precision metric for the three Q2A, QA2R and Q2AR tasks, respectively, including: 1) MUTAN [19] proposes a multimodal based visual question answering approach, which parametrizes bi-linear interactions between visual and textual representations using Tucker decomposition; 2) BERT-base [18] is a powerful pre-training based model in natural language field and is adapted for the commonsense reasoning; 3) R2C [5] encodes commonsense between sentences with LSTM; 4) DMVCR [14] trains a dynamic working memory to store the commonsense in training as well as using commonsense as prior knowledge for inference. Among them, BERT-base adopts pre-training method, while MUTAN, R2C and DMVCR are non pre-training methods.…”

“…Q2A QA2R Q2AR MUTAN [20] 44.4 32.0 14.6 BERT-base [18] 53.9 64.5 35 R2C [5] 61.9 62.8 39.1 DMVCR [14] 62. In these results, it is clear that CAN consistently outperforms other methods across all tasks and is the only method capable of handling all tasks properly.…”