A Survey of Embodied AI: From Simulators to Research Tasks

Duan, Jiafei; Jian, Samson Yu Bai; Tan, Hui Li; Zhu, Hongyuan; Tan, Cheston

doi:10.48550/arxiv.2103.04918

Cited by 5 publications

(5 citation statements)

References 120 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Artificial agents may be able to obtain these same benefits if they learn in a similar way rather than being trained from a predefined dataset of static images (Smith and Gasser 2005). To facilitate this, various simulation tools have been developed that allow artificial agents to roam and interact in a simulated world to learn about it directly (Duan et al 2021). An example of applying this to object normalization is to learn spatial invariants of object classification by approaching objects in different ways in the simulated world (Caudell et al 2011).…”

Section: Feasibility Of Architecture Componentsmentioning

confidence: 99%

A Safety Assurable Human-Inspired Perception Architecture

Salay¹,

Czarnecki²

2022

Preprint

View full text Add to dashboard Cite

Although artificial intelligence-based perception (AIP) using deep neural networks (DNN) has achieved near human level performance, its well-known limitations are obstacles to the safety assurance needed in autonomous applications. These include vulnerability to adversarial inputs, inability to handle novel inputs and non-interpretability. While research in addressing these limitations is active, in this paper, we argue that a fundamentally different approach is needed to address them. Inspired by dual process models of human cognition, where Type 1 thinking is fast and non-conscious while Type 2 thinking is slow and based on conscious reasoning, we propose a dual process architecture for safe AIP. We review research on how humans address the simplest non-trivial perception problem, image classification, and sketch a corresponding AIP architecture for this task. We argue that this architecture can provide a systematic way of addressing the limitations of AIP using DNNs and an approach to assurance of human-level performance and beyond. We conclude by discussing what components of the architecture may already be addressed by existing work and what remains future work.

show abstract

Section: Feasibility Of Architecture Componentsmentioning

confidence: 99%

A Safety Assurable Human-Inspired Perception Architecture

Salay¹,

Czarnecki²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, those works primarily focused on human intuitions of recognizing or predicting motions. But with the advancement and rise of deep learning, computer graphics, and embodied AI [22,23,24], there has been a paradigm shift towards generating synthetic datasets that range from simple 2D cartoons [25,26,27] to realistic interaction in 3D environments [13,28,14,29,30,15], all with the aim to explore machine perception of physics and causal reasoning on a deeper level. However, only datasets from CLEVRER [15], CoPhy [13], and CATER [14] are most relevant to our work.…”

Section: Related Workmentioning

confidence: 99%

SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments

Duan¹,

Jian²,

Tan³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent advancements in deep learning, computer vision, and embodied AI have given rise to synthetic causal reasoning video datasets. These datasets facilitate the development of AI algorithms that can reason about physical interactions between objects. However, datasets thus far have primarily focused on elementary physical events such as rolling or falling. There is currently a scarcity of datasets that focus on the physical interactions that humans perform daily with objects in the real world. To address this scarcity, we introduce SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments. The SPACE simulator allows us to generate the SPACE dataset, a synthetic video dataset in a 3D environment, to systematically evaluate physics-based models on a range of physical causal reasoning tasks. Inspired by daily object interactions, the SPACE dataset comprises videos depicting three types of physical events: containment, stability and contact. These events make up the vast majority of the basic physical interactions between objects. We then further evaluate it with a state-of-the-art physics-based deep model and show that the SPACE dataset improves the learning of intuitive physics with an approach inspired by curriculum learning. Repository: https://github.com/ jiafei1224/SPACE

show abstract

“…The visual quality of the rendered scenes depends both on the features of the 3D engine and on the design of the 3D models that are shared together with the environment itself. Several different tasks are studied using these 3D simulators, such as generic robot navigation, visual recognition, visual QA -see [22] and references therein. Some environments are developed in the context of open projects that might benefit from the contributions of large communities [2], [7], [8], while others are based on closed source solutions [4].…”

Section: A 3d Virtual Environmentsmentioning

confidence: 99%

Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects

Meloni¹,

Tiezzi²,

Pasqualini³

et al. 2021

Preprint

View full text Add to dashboard Cite

In the last few years, the scientific community showed a remarkable and increasing interest towards 3D Virtual Environments, training and testing Machine Learning-based models in realistic virtual worlds. On one hand, these environments could also become a mean to study the weaknesses of Machine Learning algorithms, or to simulate training settings that allow Machine Learning models to gain robustness to 3D adversarial attacks. On the other hand, their growing popularity might also attract those that aim at creating adversarial conditions to invalidate the benchmarking process, especially in the case of public environments that allow the contribution from a large community of people. Most of the existing Adversarial Machine Learning approaches are focused on static images, and little work has been done in studying how to deal with 3D environments and how a 3D object should be altered to fool a classifier that observes it. In this paper, we study how to craft adversarial 3D objects by altering their textures, using a tool chain composed of easily accessible elements. We show that it is possible, and indeed simple, to create adversarial objects using off-the-shelf limited surrogate renderers that can compute gradients with respect to the parameters of the rendering process, and, to a certain extent, to transfer the attacks to more advanced 3D engines. We propose a saliency-based attack that intersects the two classes of renderers in order to focus the alteration to those texture elements that are estimated to be effective in the target engine, evaluating its impact in popular neural classifiers.

show abstract

A Survey of Embodied AI: From Simulators to Research Tasks

Cited by 5 publications

References 120 publications

A Safety Assurable Human-Inspired Perception Architecture

A Safety Assurable Human-Inspired Perception Architecture

SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments

Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects

Contact Info

Product

Resources

About