Continuous Visual World Modeling for Autonomous Robot Manipulation

İnceoğlu, Arda; Koc, Cagatay; Kanat, Besim Ongun; Ersen, Mustafa; Sariel, Sanem

doi:10.1109/tsmc.2017.2787482

Cited by 17 publications

(10 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Moreover, instead of extracting temporal features with HMMs and temporal pyramid pooling [34] to capture unexpected trends in the data for a fixed amount of time steps, we employ self-attention enabled LSTMs to capture anomaly indicators that may be observed long before the anomaly occurrences. In our previous work [6], we present a symbolic-level anomaly identification method that processes the outputs of a visual scene modeling system [35], proprioceptive sensors and auditory data to identify anomalies with preprocessed hand-crafted features. In this study, we extend it by presenting a three stream anomaly identification framework that extracts lowlevel features from 2D images directly without considering high-level symbolic domain symbols which does not require any hand-crafted feature engineering effort.…”

Section: Literature Reviewmentioning

confidence: 99%

CLUE-AI: A Convolutional Three-stream Anomaly Identification Framework for Robot Manipulation

Altan¹,

Sariel²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Robot safety has been a prominent research topic in recent years since robots are more involved in daily tasks. It is crucial to devise the required safety mechanisms to enable service robots to be aware of and react to anomalies (i.e., unexpected deviations from intended outcomes) that arise during the execution of these tasks. Detection and identification of these anomalies is an essential step towards fulfilling these requirements. Although several architectures are proposed for anomaly detection, identification is not yet thoroughly investigated. This task is challenging since indicators may appear long before anomalies are detected. In this paper, we propose a ConvoLUtional threEstream Anomaly Identification (CLUE-AI) framework to address this problem. The framework fuses visual, auditory and proprioceptive data streams to identify everyday object manipulation anomalies. A stream of 2D images gathered through an RGB-D camera placed on the head of the robot is processed within a self-attention enabled visual stage to capture visual anomaly indicators. The auditory modality provided by the microphone placed on the robot's lower torso is processed within a designed convolutional neural network (CNN) in the auditory stage. Last, the force applied by the gripper and the gripper state are processed within a CNN to obtain proprioceptive features. These outputs are then combined with a late fusion scheme. Our novel three-stream framework design is analyzed on everyday object manipulation tasks with a Baxter humanoid robot in a semi-structured setting. The results indicate that the framework achieves an f-score of 94% outperforming the other baselines in classifying anomalies that arise during runtime.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

CLUE-AI: A Convolutional Three-stream Anomaly Identification Framework for Robot Manipulation

Altan¹,

Sariel²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The templates of the objects are stored in the knowledge base (KB) of the robot and during the execution, point clouds from the scene are compared with the templates in the knowledge base of the robot. If a point cloud is matched with a template, the object is recognized with a similarity measure [37].…”

Section: Visionmentioning

confidence: 99%

“…Clusters that are larger or smaller than a predefined threshold are discarded. Note that the segments clustered with the segmentation algorithm are classified as unknown objects, these segments do not have an object type [37].…”

Section: Visionmentioning

confidence: 99%

“…All these vision sources are fused together to construct the world model consistently by the Violet interpreter system [37]. Violet continually monitors the scene and fuses the data in order to update the world model of the robot.…”

Section: Visionmentioning

confidence: 99%

“…After gathering these data, they are processed and fused. First, visual data are interpreted with Violet [37], and the world model is constructed. Then, the auditory data are classified if a sound is received with the method that is explained in Section 4.1.1.…”

Section: Lstms For Anomaly Identificationmentioning

confidence: 99%

See 2 more Smart Citations

What went wrong?: Identification of Everyday Object Manipulation Anomalies

Altan¹,

Sariel²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Extending the abilities of service robots is important for expanding what they can achieve in everyday manipulation tasks. On the other hand, it is also essential to ensure them to determine what they can not achieve in certain cases due to either anomalies or permanent failures during task execution. Robots need to identify these situations, and reveal the reasons behind these cases to overcome and recover from them. In this paper, we propose and analyze a Long Short-Term Memories-based (LSTM-based) awareness approach to reveal the reasons behind an anomaly case that occurs during a manipulation episode in an unstructured environment. The proposed method takes into account the real-time observations of the robot by fusing visual, auditory and proprioceptive sensory modalities to achieve this task. We also provide a comparative analysis of our method with Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs). The symptoms of anomalies are first learned from a given training set, then they can be classified in real-time based on the learned models. The approaches are evaluated on a Baxter robot executing object manipulation scenarios. The results indicate that the LSTM-based method outperforms the other methods with a 0.94 classification rate in revealing causes of anomalies in case of an unexpected deviation.

show abstract

What went wrong? Identification of everyday object manipulation anomalies

Altan

Sariel

2021

Intel Serv Robotics

View full text Add to dashboard Cite

Continuous Visual World Modeling for Autonomous Robot Manipulation

Cited by 17 publications

References 35 publications

CLUE-AI: A Convolutional Three-stream Anomaly Identification Framework for Robot Manipulation

CLUE-AI: A Convolutional Three-stream Anomaly Identification Framework for Robot Manipulation

What went wrong?: Identification of Everyday Object Manipulation Anomalies

What went wrong? Identification of everyday object manipulation anomalies

Contact Info

Product

Resources

About