Long-term robot manipulation task planning with scene graph and semantic knowledge

Abstract

Purpose Autonomous robots must be able to understand long-term manipulation tasks described by humans and perform task analysis and planning based on the current environment in a variety of scenes, such as daily manipulation and industrial assembly. However, both classical task and motion planning algorithms and single data-driven learning planning methods have limitations in practicability, generalization and interpretability. The purpose of this work is to overcome the limitations of the above methods and achieve generalized and explicable long-term robot manipulation task planning. Design/methodology/approach The authors propose a planning method for long-term manipulation tasks that combines the advantages of existing methods and the prior cognition brought by the knowledge graph. This method integrates visual semantic understanding based on scene graph generation, regression planning based on deep learning and multi-level representation and updating based on a knowledge base. Findings The authors evaluated the capability of this method in a kitchen cooking task and tabletop arrangement task in simulation and real-world environments. Experimental results show that the proposed method has a significantly improved success rate compared with the baselines and has excellent generalization performance for new tasks. Originality/value The authors demonstrate that their method is scalable to long-term manipulation tasks with varying complexity and visibility. This advantage allows their method to perform better in new manipulation tasks. The planning method proposed in this work is meaningful for the present robot manipulation task and can be intuitive for similar high-level robot planning.

Semantic Representation of Robot Manipulation with Knowledge Graph

“…The experimental tasks are defined with reference to previous work on robotic manipulation task planning [ 33 ]. Assuming a fixed scene and a given manipulation task, if prior manipulative knowledge regarding the performance or demonstration of the task is available in the knowledge base, we retrieve the template and instance modules from the knowledge base, call the corresponding action sequences and parameters, and transfer them to the robot for execution.…”

Boosting visual servoing performance through RGB-based methods

“…With the advancements in computer vision and machine learning techniques, robots can now recognize, interpret and make decisions based on the perception information gathered (Qiao et al , 2022). This has resulted in the widespread use of robot perception in various applications, including the navigation of mobile robots, providing context-awareness for service robots (Miao et al , 2023), robot arm manipulation (Lin and Wang, 2021), manufacturing (Wan et al , 2022; Zeng et al , 2018), mobile robots (Qiu et al , 2019) and transportation guidance for logistic (Bloss, 2011).…”

Robot Cooking—Transferring Observations into a Planning Language: An Automated Approach in the Field of Cooking

“…Motion planning is a challenging task, since collision-free motion has to be achieved in a complex 3D environment, while also taking into account required contact with objects, including controlled intrusion (e.g., when cutting) [13,17]. In addition to motion planning, there must be efficient task planning that can divide the given recipe into individual tasks and orchestrate their execution [18,19].…”