What Action Causes This? Towards Naive Physical Action-Effect Prediction

Abstract

Despite recent advances in knowledge representation, automated reasoning, and machine learning, artificial agents still lack the ability to understand basic actioneffect relations regarding the physical world, for example, the action of cutting a cucumber most likely leads to the state where the cucumber is broken apart into smaller pieces. If artificial agents (e.g., robots) ever become our partners in joint tasks, it is critical to empower them with such action-effect understanding so that they can reason about the state of the world and plan for actions. Towards this goal, this paper introduces a new task on naive physical action-effect prediction, which addresses the relations between concrete actions (expressed in the form of verbnoun pairs) and their effects on the state of the physical world as depicted by images. We collected a dataset for this task and developed an approach that harnesses web image data through distant supervision to facilitate learning for action-effect prediction. Our empirical results have shown that web data can be used to complement a small number of seed examples (e.g., three examples for each action) for model learning. This opens up possibilities for agents to learn physical action-effect relations for tasks at hand through communication with humans with a few examples.

Beyond the Self: Using Grounded Affordances to Interpret and Describe Others’ Actions

“…In robotics and cognitive systems research, both objectdirected action recognition in external agents [32] and the incorporation of language in human-robot systems [33], [34] have received ample attention, for example using the concept of intuitive physics [35], [36] to be able to predict outcomes from real or simulated interactions with objects. A growing interest is devoted to robots that learn new cognitive skills and improve their capabilities by interacting autonomously with the surrounding environment.…”

Weakly Supervised Multilingual Causality Extraction from Wikipedia

“…The nounphrase type, e.g., global warming → malaria epidemic, has mostly been addressed by RE methods, as we discuss in §5.2. The verb-phrase type, e.g., get fired → live on unemployment insurance, has been extracted by various methods (Ning et al, 2018;Gao et al, 2018;Rehbein and Ruppenhofer, 2017;Hashimoto et al, 2015Hashimoto et al, , 2014Hashimoto et al, , 2012Do et al, 2011;Riaz and Girju, 2010;Abe et al, 2008). The clause type, e.g., I hid the car key → She's mad, has also been studied (Dunietz et al, 2017).…”

“…Another standpoint of classifying causality extraction is the information source, e.g., newspapers (Khoo et al, 1998), the web , parallel corpora (Hidey and McKeown, 2016), 12 images (Gao et al, 2018), and videos (Fire and Zhu, 2016). We used Wikipedia articles in multiple languages because they tend to be more credible than other sources, and because they allowed us to exploit multilingual redundancy.…”

“…Although many methods have been proposed for causality extraction from text (Ning et al, 2018;Gao et al, 2018;Rehbein and Ruppenhofer, 2017;Dunietz et al, 2017;Hidey and McKeown, 2016;Zhao et al, 2016;Hashimoto et al, 2014Hashimoto et al, , 2012Do et al, 2011;Riaz and Girju, 2010;Abe et al, 2008), they have rarely addressed three issues that are important for constructing a causality knowledge base (CKB). First, we should be able to verify extracted causalities, so that the CKB can sustain the credibility of its information.…”

Commonsense Justification for Action Explanation

“…To generate this list, we mix four distracting verbs with the ground-truth action verb plus a default Other. Most of the distracting verbs come from the concrete action verbs made available by (Gao et al, 2018). We first manually filtered out the verbs which have the same meaning with the ground-truth verb.…”