Multi-modal Predicate Identification using Dynamically Learned Robot Controllers

Amiri, Saeid; Wei, Suhua; Zhang, Shiqi; Sinapov, Jivko; Thomason, Jesse; Stone, Peter

doi:10.24963/ijcai.2018/645

Cited by 19 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address this, some work has focused on learning to optimize the exploratory behavior as to minimize the number of explorations needed to identify the object (Fishel and Loeb, 2012 ). Other research has proposed learning object exploration policies when attempting to identify whether a set of categories apply to an object (Amiri et al, 2018 ). In addition, methods have also been proposed to select which behaviors to be performed when learning a model for a given category based on its semantic relationship to the categories that are already known (Thomason et al, 2018 ).…”

Section: Related Workmentioning

confidence: 99%

A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

et al. 2020

Self Cite

View full text Add to dashboard Cite

From an early age, humans learn to develop an intuition for the physical nature of the objects around them by using exploratory behaviors. Such exploration provides observations of how objects feel, sound, look, and move as a result of actions applied on them. Previous works in robotics have shown that robots can also use such behaviors (e.g., lifting, pressing, shaking) to infer object properties that camera input alone cannot detect. Such learned representations are specific to each individual robot and cannot currently be transferred directly to another robot with different sensors and actions. Moreover, sensor failure can cause a robot to lose a specific sensory modality which may prevent it from using perceptual models that require it as input. To address these limitations, we propose a framework for knowledge transfer across behaviors and sensory modalities such that: (1) knowledge can be transferred from one or more robots to another, and, (2) knowledge can be transferred from one or more sensory modalities to another. We propose two different models for transfer based on variational auto-encoders and encoder-decoder networks. The main hypothesis behind our approach is that if two or more robots share multi-sensory object observations of a shared set of objects, then those observations can be used to establish mappings between multiple features spaces, each corresponding to a combination of an exploratory behavior and a sensory modality. We evaluate our approach on a category recognition task using a dataset in which a robot used 9 behaviors, coupled with 4 sensory modalities, performed multiple times on 100 objects. The results indicate that sensorimotor knowledge about objects can be transferred both across behaviors and across sensory modalities, such that a new robot (or the same robot, but with a different set of sensors) can bootstrap its category recognition models without having to exhaustively explore the full set of objects.

show abstract

Section: Related Workmentioning

confidence: 99%

A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Intelligent agents need the capabilities of both reasoning about declarative knowledge, and probabilistic planning toward achieving long-term goals. A variety of algorithms have been developed to integrate commonsense reasoning and probabilistic planning (Hanheide et al 2017;Zhang and Stone 2015;Sridharan et al 2019;Chitnis et al 2018;Zhang et al 2017;Amiri et al 2018;Veiga et al 2019), and some of them, such as (Sridharan et al 2019) and (Amiri et al 2018), also include non-deterministic dynamic laws for observations. Although the algorithms use very different computational paradigms for representing and reasoning with human knowledge (e.g., logics, probabilities, graphs, etc), they all share the goal of leveraging declarative knowledge to improve the performance in probabilistic planning.…”

Section: Related Workmentioning

confidence: 99%

“…Depending on the observability of world states, icorpp uses either Markov Decision Processes (MDPs) (Puterman 2014) or Partially Observable MDPs (POMDPs) (Kaelbling et al 1998) for probabilistic planning. As a result, icorpp has been applied to robot navigation, dialog system, and manipulation tasks (Zhang et al 2017;Amiri et al 2018). In this work, we develop a unified representation and a corresponding implementation for icorpp, where the entire reasoning and planning system can be encoded using a single program, and practitioners are completely shielded from the technical details of formulating and solving (PO)MDPs.…”

Section: Related Workmentioning

confidence: 99%

Bridging Commonsense Reasoning and Probabilistic Planning via a Probabilistic Action Language

Wang

Zhang

Lee³

2019

Theory and Practice of Logic Programming

Self Cite

View full text Add to dashboard Cite

To be responsive to dynamically changing real-world environments, an intelligent agent needs to perform complex sequential decision-making tasks that are often guided by commonsense knowledge. The previous work on this line of research led to the framework called interleaved commonsense reasoning and probabilistic planning (icorpp), which used P-log for representing commmonsense knowledge and Markov Decision Processes (MDPs) or Partially Observable MDPs (POMDPs) for planning under uncertainty. A main limitation of icorpp is that its implementation requires non-trivial engineering efforts to bridge the commonsense reasoning and probabilistic planning formalisms. In this paper, we present a unified framework to integrate icorpp’s reasoning and planning components. In particular, we extend probabilistic action language pBC+ to express utility, belief states, and observation as in POMDP models. Inheriting the advantages of action languages, the new action language provides an elaboration tolerant representation of POMDP that reflects commonsense knowledge. The idea led to the design of the system pbcplus2pomdp, which compiles a pBC+ action description into a POMDP model that can be directly processed by off-the-shelf POMDP solvers to compute an optimal policy of the pBC+ action description. Our experiments show that it retains the advantages of icorpp while avoiding the manual efforts in bridging the commonsense reasoner and the probabilistic planner.

show abstract

“…ITRS has been evaluated using two datasets: one dataset, called CY101, contains 101 objects with ten exploratory behaviors and seven types of sensory modalities [36]; and the other, called ISPY32, includes 32 objects with eight behaviors and six types of modalities [39]. Compared with existing methods from the RAL literature [2,41], ITRS reduces the overall cost of exploration in the long term while reaching a higher accuracy of attribute identification.…”

Section: Introductionmentioning

confidence: 99%

Planning Multimodal Exploratory Actions for Online Robot Attribute Learning

Zhang

Sinapov

Zhang

2021

Robotics: Science and Systems XVII

Self Cite

View full text Add to dashboard Cite

Robots frequently need to perceive object attributes, such as "red," "heavy," and "empty," using multimodal exploratory actions, such as "look," "lift," and "shake." Robot attribute learning algorithms aim to learn an observation model for each perceivable attribute given an exploratory action. Once the attribute models are learned, they can be used to identify attributes of new objects, answering questions, such as "Is this object red and empty?" Attribute learning and identification are being treated as two separate problems in the literature. In this paper, we first define a new problem called online robot attribute learning (On-RAL), where the robot works on attribute learning and attribute identification simultaneously. Then we develop an algorithm called information-theoretic reward shaping (ITRS) that actively addresses the trade-off between exploration and exploitation in On-RAL problems. ITRS was compared with competitive robot attribute learning baselines, and experimental results demonstrate ITRS' superiority in learning efficiency and identification accuracy. 1

show abstract

Multi-modal Predicate Identification using Dynamically Learned Robot Controllers

Cited by 19 publications

References 0 publications

A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

Bridging Commonsense Reasoning and Probabilistic Planning via a Probabilistic Action Language

Planning Multimodal Exploratory Actions for Online Robot Attribute Learning

Contact Info

Product

Resources

About