Joint Video and Text Parsing for Understanding Events and Answering Queries

Abstract

We propose a framework for parsing video and text jointly for understanding events and answering user queries. Our framework produces a parse graph that represents the compositional structures of spatial information (objects and scenes), temporal information (actions and events) and causal information (causalities between events and fluents) in the video and text. The knowledge representation of our framework is based on a spatial-temporal-causal And-Or graph (S/T/C-AOG), which jointly models possible hierarchical compositions of objects, scenes and events as well as their interactions and mutual contexts, and specifies the prior probabilistic distribution of the parse graphs. We present a probabilistic generative model for joint parsing that captures the relations between the input video/text, their corresponding parse graphs and the joint parse graph. Based on the probabilistic model, we propose a joint parsing system consisting of three modules: video parsing, text parsing and joint inference. Video parsing and text parsing produce two parse graphs from the input video and text respectively. The joint inference module produces a joint parse graph by performing matching, deduction and revision on the video and text parse graphs. The proposed framework has the following objectives: Firstly, we aim at deep semantic parsing of video and text that goes beyond the traditional bag-of-words approaches; Secondly, we perform parsing and reasoning across the spatial, temporal and causal dimensions based on the joint S/T/C-AOG representation; Thirdly, we show that deep joint parsing facilitates subsequent applications such as generating narrative text descriptions and answering queries in the forms of who, what, when, where and why. We empirically evaluated our system based on comparison against groundtruth as well as accuracy of query answering and obtained satisfactory results.

A Principled Framework for General Adaptive Social Robotics

“…We have shown how the learning of scripts, which is central to many of the processes described, can capitalize on the current state of the art in computer vision -the learning of possible sequences of behavior observed in an environment and encoding of the knowledge in the form of AND-OR graphs (Gupta et al, 2009;Si et al, 2011;Pei et al, 2011;Tu et al, 2014) which can in turn be encoded as scripts. In addition, rapid effective causal learning lies at the heart of learning scripts rapidly (Ho, 2014;Ho, 2016a;Ho & Liausvia, 2013, 2014.…”

“…With the advent of computer vision and other sensing technologies, scripts can be learned through visual observation (or through other sensory modalities). Recently, there had been some work done in using computer vision to observe a scene filled with (human and other) activities and construct some kind of AND-OR graphs that capture the possible sequences of activities that can take place (Gupta, Srinivasan, Shi & Davis, 2009;Si, Pei, Yao & Zhu, 2011;Pei, Jia & Zhu, 2011;Tu, Meng, Lee, Choe & Zhu. 2014).…”

“…In the spirit of the AND-OR graph characterization of learned sequences of activities (Gupta et al, 2009;Si et al, 2011;Pei et al, 2011;Tu et al, 2014), in Figure 8 we show a part of the AND-OR graph that could have been learned from the home environment of Figure 1 using the computer vision techniques mentioned above. Each of these tracks of activities could be organized into a script like that of Figure 7.…”

“…This knowledge is not built-in -the robot has earlier learned through observation that these devices could alleviate boredom by observing the causal connection between the operating of the devices involved and the reactions of the human. And as mentioned above, this learning could take place with a small number of observational instances through visual observation (Gupta et al, 2009;Si et al, 2011;Pei et al, 2011;Tu et al, 2014).…”

Measuring Machine Intelligence Through Visual Question Answering

“…That is, its evaluation shouldn't be as hard as the task itself, and it must not be solvable using shortcuts or cheats. To solve these two problems we propose the task of visual question answering (VQA) (Antol et al 2015;Geman et al 2015;Malinowski and Fritz 2014;Tu et al 2014;Bigham et al 2010;Gao et al 2015). The task of VQA requires a machine to answer a natural language question about an image as shown in figure 2.…”