Adaptive human-robot teaming through integrated symbolic and subsymbolic artificial intelligence: preliminary results

Handelman, David A.; Rivera, Corban G.; Amant, Robert St.; Holmes, Emma A.; Badger, Andrew; Yeh, Bryanna

doi:10.1117/12.2618686

Cited by 5 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our shared control strategy, the participant and the bimanual manipulators are viewed as a human-robot team, and the goal is to enable the human to control a minimal set of key DOF to maximize task performance while minimizing human workload. It is a form of adjustable autonomy in which the robot nominally knows how to perform a task, and human input is used to guide and customize robot behavior (Handelman et al, 1990 , 2022 ; National Academies of Sciences, Engineering, and Medicine, 2022 ).…”

Section: Methodsmentioning

confidence: 99%

Shared Control of Bimanual Robotic Limbs With a Brain-Machine Interface for Self-Feeding

et al. 2022

Self Cite

View full text Add to dashboard Cite

Advances in intelligent robotic systems and brain-machine interfaces (BMI) have helped restore functionality and independence to individuals living with sensorimotor deficits; however, tasks requiring bimanual coordination and fine manipulation continue to remain unsolved given the technical complexity of controlling multiple degrees of freedom (DOF) across multiple limbs in a coordinated way through a user input. To address this challenge, we implemented a collaborative shared control strategy to manipulate and coordinate two Modular Prosthetic Limbs (MPL) for performing a bimanual self-feeding task. A human participant with microelectrode arrays in sensorimotor brain regions provided commands to both MPLs to perform the self-feeding task, which included bimanual cutting. Motor commands were decoded from bilateral neural signals to control up to two DOFs on each MPL at a time. The shared control strategy enabled the participant to map his four-DOF control inputs, two per hand, to as many as 12 DOFs for specifying robot end effector position and orientation. Using neurally-driven shared control, the participant successfully and simultaneously controlled movements of both robotic limbs to cut and eat food in a complex bimanual self-feeding task. This demonstration of bimanual robotic system control via a BMI in collaboration with intelligent robot behavior has major implications for restoring complex movement behaviors for those living with sensorimotor deficits.

show abstract

Section: Methodsmentioning

confidence: 99%

Shared Control of Bimanual Robotic Limbs With a Brain-Machine Interface for Self-Feeding

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our human-robot teaming research is focused on improving mission outcomes through resilience and adaptation based on the emulation of human skill acquisition [6][7][8][9]. The goal is to combine human-directed instruction and robot-initiated discovery by integrating symbolic AI and neural network learning to produce explainable and trusted robot behavior, adjustable autonomy, and adaptive human-robot teaming.…”

Section: A Neuro-symbolic Approach To Adaptive Human-robot Teamingmentioning

confidence: 99%

“…We want this knowledge to be easily understood at a glance by humans, and to optionally mirror known good strategies for performing tasks. We use text-based Hierarchical Task Descriptions (HTDs) to specify plans of action [6,11]. Depicted in Figure 3, HTDs support simultaneous and sequential actions through the use of "and" and "then" keywords, respectively, and encode behavior trees [12] that are executed as finite state machines.…”

Section: Technical Approach 21 Task Knowledge Acquisition and Executi...mentioning

confidence: 99%

“…Figure 5 shows formation following between a leader and two robots as a component behavior we use in our teaming scenarios. For our current investigations we utilize native localization of the HoloLens and the robot following co-registration of device reference frames [6].…”

Section: Technical Approach 21 Task Knowledge Acquisition and Executi...mentioning

confidence: 99%

“… With regard to adaptive teaming, we are exploring how to enable the humans to modify the symbolic plan of action (task description) in the field using verbal and/or written commands, such as starting with alternating bounding and changing to successive bounding, adding subtasks within an existing task to customize the behavior, or modifying the behavior based on unanticipated situations.  We are studying how robot exploration behaviors learned through reinforcement learning and tunable by reward function parameters such as search speed, stealthiness and distance [6] can be used to support effective maneuver including world model building and estimation of the presence of opposing forces in the area.…”

Section: Task 3: Successive Bounding Overwatchmentioning

confidence: 99%

See 2 more Smart Citations

Multi-agent playbook for human-robot teaming

Handelman

Holmes

Badger

et al. 2023

Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications V

View full text Add to dashboard Cite

Effective human-robot teaming requires human and robot teammates to share a common understanding of the goals of their collaboration. Ideally, a complex task can be broken into smaller components to be performed by team members with defined roles, and the plan of action and assignment of roles can be changed on the fly to accommodate unanticipated situations. In this paper we describe research on adaptive human-robot teaming that uses a playbook approach to team behavior to bootstrap multi-agent collaboration. The goal is to leverage known good strategies for accomplishing tasks, such as from training and operating manuals, to enable humans and robots to "be on the same page" and work from a common knowledge base. Simultaneous and sequential actions are specified using hierarchical text-based plans and executed as behavior trees using finite state machines. We describe a real-time implementation that supports sharing of task status updates through distributed message passing. Tasks related to human-robot teaming for exploration and reconnaissance are explored with teams comprising humans wearing augmented reality headsets and quadruped robots. It is anticipated that shared task knowledge provided by multi-agent playbooks will enable humans and robots to track and predict teammate behavior and promote team transparency, accountability and trust.

show abstract