Efficient Task/Motion Planning for a Dual-arm Robot from Language Instructions and Cooking Images

Takata, Kota; Kiyokawa, Takuya; Ramirez-Alpizar, Ixchel G.; Yamanobe, Natsuki; Wan, Weiwei; Harada, Kensuke

doi:10.1109/iros47612.2022.9981280

Cited by 12 publications

(4 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bimanual Robot Manipulation. There is a rich set of literature on bimanual robot manipulation [24], [25], [26], [27], [28], [29], [30]. These works address important problems in dual-arm coordination with a focus on coordinated control and collision avoidance.…”

Section: Related Workmentioning

confidence: 99%

LEMMA: Learning Language-Conditioned Multi-Robot Manipulation

Gong,

Gao,

Gao

et al. 2023

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Complex manipulation tasks often require robots with complementary capabilities to collaborate. We introduce a benchmark for LanguagE-Conditioned Multi-robot MAnipulation (LEMMA) focused on task allocation and longhorizon object manipulation based on human language instructions in a tabletop setting. LEMMA features 8 types of procedurally generated tasks with varying degree of complexity, some of which require the robots to use tools and pass tools to each other. For each task, we provide 800 expert demonstrations and human instructions for training and evaluations. LEMMA poses greater challenges compared to existing benchmarks, as it requires the system to identify each manipulator's limitations and assign sub-tasks accordingly while also handling strong temporal dependencies in each task. To address these challenges, we propose a modular hierarchical planning approach as a baseline. Our results highlight the potential of LEMMA for developing future language-conditioned multi-robot systems.

show abstract

Section: Related Workmentioning

confidence: 99%

LEMMA: Learning Language-Conditioned Multi-Robot Manipulation

Gong,

Gao,

Gao

et al. 2023

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…On the other hand, we have proposed the framework on task and motion planning of dual-arm manipulator from a cooking recipe composed of food image and cooking instructions. In our previous research [30], we explained the entire structure of robot motion planning and detailed the recognition of ingredients from the food image and coordination of two arms. This paper focuses on the motion planning framework based on the graph structure, which solves the mentioned problem of planning the robot's cooking task from a cooking recipe.…”

Section: Related Workmentioning

confidence: 99%

Graph-Based Framework on Bimanual Manipulation Planning from Cooking Recipe

et al. 2022

View full text Add to dashboard Cite

It is difficult to effectively operate a dual-arm robot using only the information written in a cooking recipe. To cope with this problem, this paper proposes a graph-based approach on bimanual cooking motion planning from a cooking recipe. In our approach, we first decompose the cooking recipe into graph elements. Then, we try to connect the graph elements taking into account the attributes of the input/output nodes. If two graph elements cannot be connected to each other, we search for a graph element that can be inserted between them from a motion database. Since the constructed graph includes the whole sequence of the robot’s motions to perform the cooking task, we can generate a task sequence of a dual-arm manipulator simultaneously performing two different tasks by using two arms. Through experimental study, we show that it is possible to generate robot motions from a cooking recipe and perform the cooking motions while simultaneously moving the left and right arms.

show abstract

“…A. Functional Object-Oriented Network FOON and related knowledge graphs have been used in many tasks for robots, such as robotic cooking [6] and furniture assembly [7], [8], [9]. The one used here is a knowledge graph constructed through manual annotation of video demonstrations.…”

Section: Introductionmentioning

confidence: 99%

Approximate Task Tree Retrieval in a Knowledge Network for Robotic Cooking

Sakib

Paulius

Sun

2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Task planning for robotic cooking involves generating a sequence of actions for a robot to prepare a meal successfully. This paper introduces a novel task tree generation pipeline producing correct planning and efficient execution for cooking tasks. Our method first uses a large language model (LLM) to retrieve recipe instructions and then utilizes a fine-tuned GPT-3 to convert them into a task tree, capturing sequential and parallel dependencies among subtasks. The pipeline then mitigates the uncertainty and unreliable features of LLM outputs using task tree retrieval. We combine multiple LLM task tree outputs into a graph and perform a task tree retrieval to avoid questionable nodes and high-cost nodes to improve planning correctness and improve execution efficiency. Our evaluation results show its superior performance compared to previous works in task planning accuracy and efficiency.

show abstract

Efficient Task/Motion Planning for a Dual-arm Robot from Language Instructions and Cooking Images

Cited by 12 publications

References 26 publications

LEMMA: Learning Language-Conditioned Multi-Robot Manipulation

LEMMA: Learning Language-Conditioned Multi-Robot Manipulation

Graph-Based Framework on Bimanual Manipulation Planning from Cooking Recipe

Approximate Task Tree Retrieval in a Knowledge Network for Robotic Cooking

Contact Info

Product

Resources

About