MERLIN a Cognitive Architecture for Service Robots

González-Santamarta, Miguel Á.; Rodríguez-Lera, Francisco J.; Álvarez-Aparicio, Claudia; Guerrero‐Higueras, Ángel Manuel; Fernández-Llamas, Camino

doi:10.3390/app10175989

Cited by 13 publications

(7 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent works such as [1], [5], [10] and [12] have successfully incorporated LLMs and VLMs into robotic systems, allowing the LLMs to interpret and execute complex commands. Similarly, Wangchunshu et al [35], Kaiwen Zhou et al [34], and Miguel et al [9] in their work demonstrated how LLMs could be used to facilitate real-time feedback, zero-shot object navigation and cognitive learning in autonomous robots.…”

Section: Related Workmentioning

confidence: 99%

The Conversation is the Command: Interacting with Real-World Autonomous Robots Through Natural Language

Nwankwo,

Rueckert

2024

Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction

View full text Add to dashboard Cite

In recent years, autonomous agents have surged in real-world environments such as our homes, offices, and public spaces. However, natural human-robot interaction remains a key challenge. In this paper, we introduce an approach that synergistically exploits the capabilities of large language models (LLMs) and multimodal vision-language models (VLMs) to enable humans to interact naturally with autonomous robots through conversational dialogue. We leveraged the LLMs to decode the high-level natural language instructions from humans and abstract them into precise robot actionable commands or queries. Further, we utilised the VLMs to provide a visual and semantic understanding of the robot's task environment. Our results with 99.13% command recognition accuracy and 97.96% commands execution success show that our approach can enhance human-robot interaction in real-world applications. The video demonstrations of this paper can be found at https://osf.io/wzyf6 and the code is available at our repository 1 . CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); Interaction paradigms; Natural language interfaces;

show abstract

Section: Related Workmentioning

confidence: 99%

The Conversation is the Command: Interacting with Real-World Autonomous Robots Through Natural Language

Nwankwo,

Rueckert

2024

Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction

View full text Add to dashboard Cite

show abstract

“…This research uses MERLIN [ 58 ], which combines deliberative and behavioural capacities. Deliberative capacities define the characteristics of planning to infer long-term tasks.…”

Section: System Architecturementioning

confidence: 99%

“…It assumes a background approach, feeding a knowledge database and keeping the deliberative monitoring system that would interrupt the active task up to date. The architecture proposed in [ 58 ] requires to be updated by including the new vision-based module proposed in this paper in the reactive layer. As this reactive layer gathers all the sensors of the robot, with the proposed vision module, the images acquired by the robot camera are processed.…”

Section: System Architecturementioning

confidence: 99%

Vision-Based Module for Herding with a Sheepdog Robot

Castillo

Sánchez-González

Campazas-Vega

et al. 2022

Sensors

View full text Add to dashboard Cite

Livestock farming is assisted more and more by technological solutions, such as robots. One of the main problems for shepherds is the control and care of livestock in areas difficult to access where grazing animals are attacked by predators such as the Iberian wolf in the northwest of the Iberian Peninsula. In this paper, we propose a system to automatically generate benchmarks of animal images of different species from iNaturalist API, which is coupled with a vision-based module that allows us to automatically detect predators and distinguish them from other animals. We tested multiple existing object detection models to determine the best one in terms of efficiency and speed, as it is conceived for real-time environments. YOLOv5m achieves the best performance as it can process 64 FPS, achieving an mAP (with IoU of 50%) of 99.49% for a dataset where wolves (predator) or dogs (prey) have to be detected and distinguished. This result meets the requirements of pasture-based livestock farms.

show abstract

“…In [29], researchers developed TRs for offices, healthcare systems, and nursing homes. Another useful application is augmented virtual reality, which is useful to simulate the feeling of a human-robot interactive environment [30][31][32][33]. In [34], immersed virtual reality was developed to provide guidelines for user design.…”

Section: Introductionmentioning

confidence: 99%

Design of a Telepresence Robot to Avoid Obstacles in IoT-Enabled Sustainable Healthcare Systems

2023

View full text Add to dashboard Cite

In the Internet of Things (IoT) era, telepresence robots (TRs) are increasingly a part of healthcare, academia, and industry due to their enormous benefits. IoT provides a sensor-based environment in which robots receive more precise information about their surroundings. The researchers work day and night to reduce cost, duration, and complexity in all application areas. It provides tremendous benefits, such as sustainability, welfare improvement, cost-effectiveness, user-friendliness, and adaptability. However, it faces many challenges in making critical decisions during motion, which requires a long training period and intelligent motion planning. These include obstacle avoidance during movement, intelligent control in hazardous situations, and ensuring the right measurements. Following up on these issues requires a sophisticated control design and a secure communication link. This paper proposes a control design to normalize the integration process and offer an auto-MERLIN robot with cognitive and sustainable architecture. A control design is proposed through system identification and modeling of the robot. The robot control design was evaluated, and a prototype was prepared for testing in a hazardous environment. The robot was tested by considering various parameters: driving straight ahead, turning right, self-localizing, and receiving commands from a remote location. The maneuverability, controllability, and stability results show that the proposed design is well-developed and cost-efficient, with a fast response time. The experimental results show that the proposed method significantly minimizes the obstacle collisions. The results confirm the employability and sustainability of the proposed design and demonstrate auto-MERLIN’s capabilities as a sustainable robot ready to be deployed in highly interactive scenarios.

show abstract

MERLIN a Cognitive Architecture for Service Robots

Cited by 13 publications

References 21 publications

The Conversation is the Command: Interacting with Real-World Autonomous Robots Through Natural Language

The Conversation is the Command: Interacting with Real-World Autonomous Robots Through Natural Language

Vision-Based Module for Herding with a Sheepdog Robot

Design of a Telepresence Robot to Avoid Obstacles in IoT-Enabled Sustainable Healthcare Systems

Contact Info

Product

Resources

About