Do You Follow?

Axelsson, Agnes; Skantze, Gabriel

doi:10.1145/3568162.3576958

Cited by 12 publications

(9 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LLMs have recently also been applied to social robots in human-robot interaction (HRI) [19,[42][43][44][45]. One example of this is the collaborative story-telling game Cre-ativeBot [42], where a child and a robot take turns creating a story.…”

Section: Large Language Models In Social Robotsmentioning

confidence: 99%

Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

Irfan

Kuoppamäki

Skantze

2023

Preprint

Self Cite

View full text Add to dashboard Cite

This work aims to provide initial guidelines towards developing companion robots with large language models (LLMs) to be part of everyday lives of older adults. Using iterative participatory design (co-design) approaches, we analyze the challenges of applying LLMs for multi-modal open-domain dialogue, deriving from older adults' (one-to-one) interactions with a personalized companion robot, built on Furhat robot with GPT-3.5. An initial study with 6 Swedish-speaking older adults (65 and older) showed that the robot frequently interrupted the users, responded slowly and repetitively, engaged in superficial conversations, and caused a barrier in the interaction due to foreign language (English). Upon incremental technical developments to address these issues, participatory design workshops were conducted with 28 Swedish-speaking older adults. While the interactions (in Swedish) were smoother, less disrupted, and more varied in topics and responses, further challenges were observed due to hallucinations and obsolete information, and disengagement cues, causing frustration, confusion, and worry.

show abstract

Section: Large Language Models In Social Robotsmentioning

confidence: 99%

Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

Irfan

Kuoppamäki

Skantze

2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Within the HRI literature, very recently a number of works, e.g., [4,11,17,31], have started to investigate and evaluate the use of adaptive and personalised robots by showing that adaptive configurations are very promising for robotic applications [39]. For example, Axelsson and Skantze [4] presented a fully automated system for building adaptive presentations for embodied agents. They evaluated the system involving 43 participants who interacted with the adaptive system, and their results demonstrated that the user preferred the adaptive system.…”

Section: Adaptation and Personalization In Hrimentioning

confidence: 99%

“…This is a well-known problem in HRI. Past works have shown how adaptive models were better performance-wise, but users -interacting with the systemusually cannot perceive the differences across adaptive system's configurations [4,17].…”

Section: Emulating Human Well-being Coach Behaviour Via Adaptive Robo...mentioning

confidence: 99%

Robotic Mental Well-being Coaches for the Workplace

Spitale

Axelsson

Güneş

2023

Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction

View full text Add to dashboard Cite

Recently, several works have explored if and how robotic coaches can promote and maintain mental well-being in different settings. However, findings from these studies revealed that these robotic coaches are not ready to be used and deployed in real-world settings due to several limitations that span from technological challenges to coaching success. To overcome these challenges, this paper presents VITA, a novel multi-modal LLM-based system that allows robotic coaches to autonomously adapt to the coachee's multi-modal behaviours (facial valence and speech duration) and deliver coaching exercises in order to promote mental well-being in adults. We identified five objectives that correspond to the challenges in the recent literature, and we show how the VITA system addresses these via experimental validations that include one in-lab pilot study (𝑁 = 4) that enabled us to test different robotic coach configurations (pre-scripted, generic, and adaptive models) and inform its design for using it in the real world, and one real-world study (𝑁 = 17) conducted in a workplace over 4 weeks. Our results show that: (i) coachees perceived the VITA adaptive and generic configurations more positively than the pre-scripted one, and they felt understood and heard by the adaptive robotic coach, (ii) the VITA adaptive robotic coach kept learning successfully by personalising to each coachee over time and did not detect any interaction ruptures during the coaching, (iii) coachees had significant mental well-being improvements via the VITA-based robotic coach practice. The code for the VITA system is openly available via https://github.com/Cambridge-AFAR/VITA-system. CCS Concepts: • Human-centered computing → Empirical studies in HCI ; HCI theory, concepts and models.

show abstract

“…While various foundation models are used in robotics for manipulation, navigation, planning, and reasoning ( Xiao et al, 2023 ), only LLMs are used in the context of conversational robots. For instance, LLMs have been used for developing conversational robots with empathetic non-verbal cues ( Lee et al, 2023 ), giving adaptive presentations ( Axelsson and Skantze, 2023 ), functioning as a receptionist ( Cherakara et al, 2023 ; Yamazaki et al, 2023 ), and supporting wellbeing of older adults ( Khoo et al, 2023 ). Khoo et al (2023) is the only study that integrated an LLM (fine-tuned GPT-3) into a companion robot for open-domain dialogue with (7) older adults, in addition to our prior work ( Irfan et al, 2023 ).…”

Section: Introductionmentioning

confidence: 99%

Recommendations for designing conversational companion robots with older adults through foundation models

Irfan,

Kuoppamäki,

Skantze

2024

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

Companion robots are aimed to mitigate loneliness and social isolation among older adults by providing social and emotional support in their everyday lives. However, older adults’ expectations of conversational companionship might substantially differ from what current technologies can achieve, as well as from other age groups like young adults. Thus, it is crucial to involve older adults in the development of conversational companion robots to ensure that these devices align with their unique expectations and experiences. The recent advancement in foundation models, such as large language models, has taken a significant stride toward fulfilling those expectations, in contrast to the prior literature that relied on humans controlling robots (i.e., Wizard of Oz) or limited rule-based architectures that are not feasible to apply in the daily lives of older adults. Consequently, we conducted a participatory design (co-design) study with 28 older adults, demonstrating a companion robot using a large language model (LLM), and design scenarios that represent situations from everyday life. The thematic analysis of the discussions around these scenarios shows that older adults expect a conversational companion robot to engage in conversation actively in isolation and passively in social settings, remember previous conversations and personalize, protect privacy and provide control over learned data, give information and daily reminders, foster social skills and connections, and express empathy and emotions. Based on these findings, this article provides actionable recommendations for designing conversational companion robots for older adults with foundation models, such as LLMs and vision-language models, which can also be applied to conversational robots in other domains.

show abstract

Do You Follow?

Cited by 12 publications

References 45 publications

Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

Robotic Mental Well-being Coaches for the Workplace

Recommendations for designing conversational companion robots with older adults through foundation models

Contact Info

Product

Resources

About