Action recognition and interpretation from virtual demonstrations

Haidu, Andrei; Beetz, Michael

doi:10.1109/iros.2016.7759439

Cited by 13 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Haidu and Beetz introduced a virtual reality-based skill learning environment for daily life operations in [ 25 ], which allows users to use Razer Hydra controllers to control virtual hands. From the many records of humans performing tasks in their environment, they are able to train (simulated) robotic agents to perform the same tasks.…”

Section: Related Workmentioning

confidence: 99%

Real-Time Learning and Recognition of Assembly Activities Based on Virtual Reality Demonstration

Zhang

Zhao

2021

Sensors

View full text Add to dashboard Cite

Teaching robots to learn through human demonstrations is a natural and direct method, and virtual reality technology is an effective way to achieve fast and realistic demonstrations. In this paper, we construct a virtual reality demonstration system that uses virtual reality equipment for assembly activities demonstration, and using the motion data of the virtual demonstration system, the human demonstration is deduced into an activity sequence that can be performed by the robot. Through experimentation, the virtual reality demonstration system in this paper can achieve a 95% correct rate of activity recognition. We also created a simulated ur5 robotic arm grasping system to reproduce the inferred activity sequence.

show abstract

Section: Related Workmentioning

confidence: 99%

Real-Time Learning and Recognition of Assembly Activities Based on Virtual Reality Demonstration

Zhang

Zhao

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Moreover, many systems lack symbolic task information, such as teleoperated robots for space operations or minimally invasive surgery. To circumvent these issues, we propose to annotate the sensor streams in a post-processing step [49], based on the reasoning mechanisms presented in Sec. 4.…”

Section: Semantic Analysis With Openeasementioning

confidence: 99%

Cognition-enabled robotic wiping: Representation, planning, execution, and interpretation

Leidner

Bartels

Bejjani

et al. 2019

Robotics and Autonomous Systems

Self Cite

View full text Add to dashboard Cite

Advanced cognitive capabilities enable humans to solve even complex tasks by representing and processing internal models of manipulation actions and their effects. Consequently, humans are able to plan the effect of their motions before execution and validate the performance afterwards. In this work, we derive an analog approach for robotic wiping actions which are fundamental for some of the most frequent household chores including vacuuming the floor, sweeping dust, and cleaning windows. We describe wiping actions and their effects based on a qualitative particle distribution model. This representation enables a robot to plan goal-oriented wiping motions for the prototypical wiping actions of absorbing, collecting and skimming. The particle representation is utilized to simulate the task outcome before execution and infer the real performance afterwards based on haptic perception. This way, the robot is able to estimate the task performance and schedule additional motions if necessary. We evaluate our methods in simulated scenarios, as well as in real experiments with the humanoid service robot Rollin' Justin.

show abstract

“…It is important in the cut & recombine method that ADT information should transfer across similar robotic systems. Hence, eight of those ADTs were acquired using the KUKA LWR arm with Schunk SDH2 gripper, the same as used in the test experiments; three ADTs were acquired using a Universal Robot Arm UR5 with Schunk WSG50 gripper (Kramberger et al, 2016); and the remaining 17 ADTs were made in simulations using a Razor Hydra device and the robotic simulator Gazebo , as described by Haidu and Beetz (2016).…”

Section: Data Structuresmentioning

confidence: 99%

Cut & recombine: reuse of robot action components based on simple language instructions

Tamošiūnaitė

Aein

Braun

et al. 2019

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

Human beings can generalize from one action to similar ones. Robots cannot do this and progress concerning information transfer between robotic actions is slow. We have designed a system that performs action generalization for manipulation actions in different scenarios. It relies on an action representation for which we perform code-snippet replacement, combining information from different actions to form new ones. The system interprets human instructions via a parser using simplified language. It uses action and object names to index action data tables (ADTs), where execution-relevant information is stored. We have created an ADT database from three different sources (KUKA LWR, UR5, and simulation) and show how a new ADT is generated by cutting and recombining data from existing ADTs. To achieve this, a small set of action templates is used. After parsing a new instruction, index-based searching finds similar ADTs in the database. Then the action template of the new action is matched against the information in the similar ADTs. Code snippets are extracted and ranked according to matching quality. The new ADT is created by concatenating code snippets from best matches. For execution, only coordinate transforms are needed to account for the poses of the objects in the new scene. The system was evaluated, without additional error correction, using 45 unknown objects in 81 new action executions, with 80% success. We then extended the method including more detailed shape information, which further reduced errors. This demonstrates that cut & recombine is a viable approach for action generalization in service robotic applications.

show abstract

Action recognition and interpretation from virtual demonstrations

Cited by 13 publications

References 9 publications

Real-Time Learning and Recognition of Assembly Activities Based on Virtual Reality Demonstration

Real-Time Learning and Recognition of Assembly Activities Based on Virtual Reality Demonstration

Cognition-enabled robotic wiping: Representation, planning, execution, and interpretation

Cut & recombine: reuse of robot action components based on simple language instructions

Contact Info

Product

Resources

About