Evaluating the quality of voice assistants’ responses to consumer health questions about vaccines: an exploratory comparison of Alexa, Google Assistant and Siri

Couvillon, Emily; Helbing, Rachel

doi:10.1136/bmjhci-2019-100075

Cited by 56 publications

(77 citation statements)

References 16 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The rubric used was adapted from 3 studies on voice assistants in healthcare [5,6,8] and the DISCERN [30] and HONcode [31,32] quality evaluation tools (Figure 1). The point system was adapted from Alagha and Helbing [6]. The rubric evaluated 5 parameters: relevance, reliability, accuracy, comprehensiveness, and user-friendliness of information provided.…”

Section: Evaluation Rubricmentioning

confidence: 99%

“…Another study by Boyd and Wilson found that Google internet searches and Google Assistant fared better than Siri for smoking cessation information, but there was room for improvement for all three voice assistants in sourcing expert content [5]. Alagha and Helbing found that Google Assistant and Siri understood consumer queries about vaccine safety and use better and provided more reliable sources than Alexa [6]. In contrast, Miner et al reported that Siri, Cortana, Google Now and S Voice were inconsistent and incomplete in their responses to queries regarding mental health, interpersonal violence and physical health [7].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation of COVID-19 Information Provided by Digital Voice Assistants

Goh

Wong

Yap

2021

International Journal of Digital Health

View full text Add to dashboard Cite

Background: Digital voice assistants are widely used for health information seeking activities during the COVID-19 pandemic. Due to the rapidly changing nature of COVID-19 information, there is a need to evaluate COVID-related information provided by voice assistants, to ensure consumers' needs are met and prevent misinformation. The objective of this study is to evaluate COVID-related information provided by the voice assistants in terms of relevance, accuracy, comprehensiveness, user-friendliness and reliability. Materials and Methods: The voice assistants evaluated were Amazon Alexa, Google Home, Google Assistant, Samsung Bixby, Apple Siri and Microsoft Cortana. Two evaluators posed COVID-19 questions to the voice assistants and evaluated responses based on relevance, accuracy, comprehensiveness, user-friendliness and reliability. Questions were obtained from the World Health Organization, governmental websites, forums and search trends. Data was analyzed using Pearson's correlation, independent samples t-tests and Wilcoxon rank-sum tests. Results: Google Assistant and Siri performed the best across all evaluation parameters with mean scores of 84.0% and 80.6% respectively. Bixby performed the worst among the smartphone-based voice assistants (65.8%). On the other hand, Google Home performed the best among the non-smartphone voice assistants (60.7%), followed by Alexa (43.1%) and Cortana (13.3%). Smartphone-based voice assistants had higher mean scores than voice assistants on other platforms (76.8% versus 39.1%, p = 0.064). Google Assistant consistently scored better than Google Home for all the evaluation parameters. A decreasing score trend from Google Assistant, Siri, Bixby, Google Home, Alexa and Cortana was observed for majority of the evaluation criteria, except for accuracy, comprehensiveness and credibility. Conclusion: Google Assistant and Apple Siri were able to provide users with relevant, accurate, comprehensive, user-friendly, and reliable information regarding COVID-19. With the rapidly evolving information on this pandemic, users need to be discerning when obtaining COVID-19 information from voice assistants.

show abstract

Section: Evaluation Rubricmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Evaluation of COVID-19 Information Provided by Digital Voice Assistants

Goh

Wong

Yap

2021

International Journal of Digital Health

View full text Add to dashboard Cite

show abstract

“…Intelligent conversational agents and virtual assistants have been used and tested for responding to health information seeking activities [8][9][10][11][12] . Recent efforts have expanded their utilities in health assessment.…”

Section: Introductionmentioning

confidence: 99%

Readiness for voice assistants to support healthcare delivery during a health crisis and pandemic

et al. 2020

View full text Add to dashboard Cite

To prevent the spread of COVID-19 and to continue responding to healthcare needs, hospitals are rapidly adopting telehealth and other digital health tools to deliver care remotely. Intelligent conversational agents and virtual assistants, such as chatbots and voice assistants, have been utilized to augment health service capacity to screen symptoms, deliver healthcare information, and reduce exposure. In this commentary, we examined the state of voice assistants (e.g., Google Assistant, Apple Siri, Amazon Alexa) as an emerging tool for remote healthcare delivery service and discussed the readiness of the health system and technology providers to adapt voice assistants as an alternative healthcare delivery modality during a health crisis and pandemic.

show abstract

“…In 2020, 27% of all web searches used Google Assistant [ 2 ], and the adoption of VA-enabled speakers (eg, Amazon Echo) is increasing, with an estimated $3.5 billion in spending in the United States by 2021 [ 3 ]. In the last 4 years, VA as a digital health tool has been evaluated for information seeking regarding a healthy lifestyle [ 4 ], addiction [ 5 ], vaccination [ 6 ], mental health, and interpersonal violence [ 7 ]. Mental health stands out as a prime candidate for VA-based digital health intervention to fulfill technology’s promise to provide adaptive and personalized care [ 8 , 9 ].…”

Section: Introductionmentioning

confidence: 99%

Clinical Advice by Voice Assistants on Postpartum Depression: Cross-Sectional Investigation Using Apple Siri, Amazon Alexa, Google Assistant, and Microsoft Cortana

Yang¹,

Lee²,

Sezgın³

et al. 2021

JMIR Mhealth Uhealth

View full text Add to dashboard Cite

Background A voice assistant (VA) is inanimate audio-interfaced software augmented with artificial intelligence, capable of 2-way dialogue, and increasingly used to access health care advice. Postpartum depression (PPD) is a common perinatal mood disorder with an annual estimated cost of $14.2 billion. Only a small percentage of PPD patients seek care due to lack of screening and insufficient knowledge of the disease, and this is, therefore, a prime candidate for a VA-based digital health intervention. Objective In order to understand the capability of VAs, our aim was to assess VA responses to PPD questions in terms of accuracy, verbal response, and clinically appropriate advice given. Methods This cross-sectional study examined four VAs (Apple Siri, Amazon Alexa, Google Assistant, and Microsoft Cortana) installed on two mobile devices in early 2020. We posed 14 questions to each VA that were retrieved from the American College of Obstetricians and Gynecologists (ACOG) patient-focused Frequently Asked Questions (FAQ) on PPD. We scored the VA responses according to accuracy of speech recognition, presence of a verbal response, and clinically appropriate advice in accordance with ACOG FAQ, which were assessed by two board-certified physicians. Results Accurate recognition of the query ranged from 79% to 100%. Verbal response ranged from 36% to 79%. If no verbal response was given, queries were treated like a web search between 33% and 89% of the time. Clinically appropriate advice given by VA ranged from 14% to 29%. We compared the category proportions using the Fisher exact test. No single VA statistically outperformed other VAs in the three performance categories. Additional observations showed that two VAs (Google Assistant and Microsoft Cortana) included advertisements in their responses. Conclusions While the best performing VA gave clinically appropriate advice to 29% of the PPD questions, all four VAs taken together achieved 64% clinically appropriate advice. All four VAs performed well in accurately recognizing a PPD query, but no VA achieved even a 30% threshold for providing clinically appropriate PPD information. Technology companies and clinical organizations should partner to improve guidance, screen patients for mental health disorders, and educate patients on potential treatment.

show abstract

Evaluating the quality of voice assistants’ responses to consumer health questions about vaccines: an exploratory comparison of Alexa, Google Assistant and Siri

Cited by 56 publications

References 16 publications

Evaluation of COVID-19 Information Provided by Digital Voice Assistants

Evaluation of COVID-19 Information Provided by Digital Voice Assistants

Readiness for voice assistants to support healthcare delivery during a health crisis and pandemic

Clinical Advice by Voice Assistants on Postpartum Depression: Cross-Sectional Investigation Using Apple Siri, Amazon Alexa, Google Assistant, and Microsoft Cortana

Contact Info

Product

Resources

About