Learning to Ground Objects for Robot Task and Motion Planning

Ding, Yan; Zhang, Xiaohan; Zhan, Xingyue; Zhang, Shiqi

doi:10.1109/lra.2022.3157566

Cited by 7 publications

(5 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some researchers focus on general methods, including designing appropriate prompts to guide the model to output the correct answer [12], and connecting the model output with the robot motions [13]. More researchers have studied the specific applications of LLMs in robot task and motion planning [14,15], navigation [16], and command control [17], among others. However, most of these studies are oriented towards the field of robotics, with limited research on the application of LLMs in HRC manipulation.…”

Section: A Decision-making Methods In Hrc Systemsmentioning

confidence: 99%

“…Recently, large language models (LLMs) have exhibited robust generalization and reasoning abilities across multiple tasks [10,11]. Specifically, their success on various robot tasks has demonstrated their potential to offer universal decision-making capabilities for HRC systems [12][13][14][15][16][17]. The emergence of LLMs presents new opportunities to robotic, making it highly significant to study their application in HRC.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Cognitive Digital Twin System of Human-Robot Collaboration Manipulation

Li,

He,

Wang

et al. 2024

Preprint

View full text Add to dashboard Cite

Multielement decision-making is crucial for the robust deployment of human-robot collaboration (HRC) systems in flexible manufacturing environments with personalized tasks and dynamic scenes. Large Language Models (LLMs) have recently demonstrated remarkable reasoning capabilities in various robotic tasks, potentially providing this ability. However, the application of LLMs to actual HRC systems necessitates the timely and comprehensive capture of real-scene information. In this study, we suggest incorporating real scene data into LLMs using digital twin (DT) technology and present a cognitive digital twin prototype system of HRC manipulation, known as HRCCogiDT. Specifically, we initially construct a scene semantic graph that encodes the geometric information of entities, spatial relationships between entities, actions of humans and robots, and collaborative activities. Subsequently, we devise a prompt that merges scene semantics with activity priori, linking the real scene with LLMs. To evaluate performance, we compile an HRC scene understanding dataset and set up a laboratory-level experimental platform. Empirical results indicate that HRC-CogiDT can swiftly perceive scene changes and make high-level decisions according to varying task requirements, such as task planning, anomaly detection, and schedule reasoning. This study provides promising implications for future applications of LLMs in robotics.

show abstract

Section: A Decision-making Methods In Hrc Systemsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Towards Cognitive Digital Twin System of Human-Robot Collaboration Manipulation

Li,

He,

Wang

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Language translation in the context of LLMs and planning involves transforming natural language instructions into structured planning languages (Wong et al 2023;Kelly et al 2023;Pan et al 2023;Xie et al 2023;Yang, Ishay, and Lee 2023;Lin et al 2023c;Sakib and Sun 2023;Yang et al 2023b;Parakh et al 2023;Yang et al 2023a;Dai et al 2023;Ding et al 2023b;Zelikman et al 2023;Xu et al 2023b;Chen et al 2023a;You et al 2023) such as PDDL, and vice versa, utilizing in-context learning techniques (Guan et al 2023). This capability effectively bridges the gap between human linguistic expression and machine-understandable formats, enhancing intuitive and efficient planning processes.…”

Section: Language Translationmentioning

confidence: 99%

“…Efforts to generate multimodal, text, and image-based goalconditioned plans are exemplified by (Lu et al 2023b). Additionally, a subset of studies in this survey investigates the fine-tuning of seq2seq, code-based language models (Pallagani et al 2022(Pallagani et al , 2023b, which are noted for their advanced Application of LLMs in Planning Language Translation ( 23) Xie et al 2023;Guan et al 2023;Chalvatzaki et al 2023;Yang, Ishay, and Lee 2023;Wong et al 2023;Kelly et al 2023;Lin et al 2023c;Sakib and Sun 2023;Yang et al 2023b;Parakh et al 2023;Dai et al 2023;Yang et al 2023a;Shirai et al 2023;Ding et al 2023b;Zelikman et al 2023;Pan et al 2023;Xu et al 2023b;Brohan et al 2023;Yang, Gaglione, and Topcu 2022;Chen et al 2023a;You et al 2023) Plan Generation (53) (Sermanet et al 2023;Li et al 2023b;Pallagani et al 2022;Silver et al 2023;Pallagani et al 2023b;Arora and Kambhampati 2023;Fabiano et al 2023;Chalvatzaki et al 2023;Gu et al 2023;Silver et al 2022;Hao et al 2023a;Lin et al 2023b;Yuan et al 2023b;Gandhi, Sadigh, and Goodman 2023;…”

Section: Plan Generationmentioning

confidence: 99%

See 1 more Smart Citation

On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)

Pallagani,

Muppasani,

Roy

et al. 2024

ICAPS

View full text Add to dashboard Cite

Automated Planning and Scheduling is among the growing areas in Artificial Intelligence (AI) where mention of LLMs has gained popularity. Based on a comprehensive review of 126 papers, this paper investigates eight categories based on the unique applications of LLMs in addressing various aspects of planning problems: language translation, plan generation, model construction, multi-agent planning, interactive planning, heuristics optimization, tool integration, and brain-inspired planning. For each category, we articulate the issues considered and existing gaps. A critical insight resulting from our review is that the true potential of LLMs unfolds when they are integrated with traditional symbolic planners, pointing towards a promising neuro-symbolic approach. This approach effectively combines the generative aspects of LLMs with the precision of classical planning methods. By synthesizing insights from existing literature, we underline the potential of this integration to address complex planning challenges. Our goal is to encourage the ICAPS community to recognize the complementary strengths of LLMs and symbolic planners, advocating for a direction in automated planning that leverages these synergistic capabilities to develop more advanced and intelligent planning systems. We aim to keep the categorization of papers updated on https://ai4society.github.io/LLM-Planning-Viz/, a collaborative resource that allows researchers to contribute and add new literature to the categorization.

show abstract