Engagement and Usability of Conversational Search – A Study of a Medical Resource Center Chatbot

Fergencs, Tamás; Meier, Florian

doi:10.1007/978-3-030-71292-1_26

Cited by 8 publications

(7 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These experiments measured usability characteristics referred to effectiveness, efficiency and satisfaction as shown in Table 1. Some of the primary studies ( [19], [26], [27], [28], [29], [30], [31], [32], [33]) consider all three of these aspects, others ( [24], [25], [34], [35], [36], [37], [38], [39], [40]) investigate efficiency and satisfaction, others again ( [41], [42], [43], [44], [45]) consider only satisfaction, whereas only one ( [46]) evaluated both effectiveness and satisfaction. Satisfaction is again the usability characteristic of most concern to researchers since it is evaluated most often.…”

Section: Related Workmentioning

confidence: 99%

Using the SOCIO Chatbot for UML Modelling: A Family of Experiments

Ren

Castro

Santos

et al. 2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Context: Recent developments in natural language processing have facilitated the adoption of chatbots in typically collaborative software engineering tasks (such as diagram modelling). Families of experiments can assess the performance of tools and processes and, at the same time, alleviate some of the typical shortcomings of individual experiments (e.g., inaccurate and potentially biased results due to a small number of participants). Objective: Compare the usability of a chatbot for collaborative modelling (i.e., SOCIO) and an online web tool (i.e., Creately). Method: We conduct a family of three experiments to evaluate the usability of SOCIO against the Creately online collaborative tool in academic settings. Results: The student participants were faster at building class diagrams using the chatbot than with the online collaborative tool and more satisfied with SOCIO. Besides, the class diagrams built using the chatbot tended to be more concise -albeit slightly less complete. Conclusion: Chatbots appear to be helpful for building class diagrams. In fact, our study has helped us to shed light on the future direction for experimentation in this field and lays the groundwork for researching the applicability of chatbots in diagramming.

show abstract

Section: Related Workmentioning

confidence: 99%

Using the SOCIO Chatbot for UML Modelling: A Family of Experiments

Ren

Castro

Santos

et al. 2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…For both chatbots, the level of reuse intention and recommendation for others stated by participants were not as high as expected. [PS25] This study aims to compare the conversational search user interface (chatbot) of a medical resource center database with its graphical search user interface in terms of user engagement and usability.…”

Section: Nomentioning

confidence: 99%

“…However, the adoption of usability scales varies a lot in chatbot usability experiments, because it mainly depends on the research goal and chatbot type.Researchers developed multiple questions for self-designed questionnaires according to their research topics and measurements. For example, researchers developed, based on the SUS questionnaire, the Voice Usability Scale for speechbased systems, as SUS does not comprehensively account for several characteristics that are unique to a voice environment[PS18].Most experiments used standardized usability scales (like Affective Slider [PS1], ResQue model [PS3], [PS19], SUS [PS6], [PS8], [PS14], [PS17], SUMI [PS13], AdjectiveRating Scale[PS18], Usefulness, Satisfaction, and Ease of use questions[PS23], User Engagement Scale[PS25]), whereas scales cited in national and international standards (SUS and SUMI) were adopted in only five experiments.The chatbot usability experiment correlates to chatbot development. In general, evaluations of chatbot usability were considered as a part of the software development process.…”

mentioning

confidence: 99%

Experimentation for Chatbot Usability Evaluation: A Secondary Study

et al. 2022

View full text Add to dashboard Cite

Interest in chatbot development is on the rise. As a usability evaluation is an essential step in chatbot development, the number of experimental studies on chatbot usability has grown as well. As a result, we think a systematic mapping study is opportune. We analyzed more than 700 sources and retrieved 28 primary studies. By aggregating the research questions and examining the characteristics and metrics used to evaluate the usability of chatbots in experiments, it is possible to identify the state of the art in chatbot usability experimentation. We conducted a systematic mapping study to identify the research questions, characteristics, and metrics used to evaluate the usability of chatbots in experiments. Most experiments adopted a within-subjects design. On the other hand, few experiments provided raw data, and only one of the identified papers was part of a family of experiments. Effectiveness, efficiency, and satisfaction are usability characteristics used to identify how well users can learn and use chatbots to achieve their goals and how satisfied users are during the interaction. Generally, the experimental results revealed that chatbots have several advantages (e.g., they provide a real-time response and they improve ease of use) and some shortcomings (e.g., natural language processing, which is rated as the weakness most in need of improvement). This research offers an overview of chatbot usability experimentation. The increasing interest in this area is very recent, as works did not start to be published until 2018. Chatbot usability experiments should be more replicable to improve the reliability and transparency of the experimental results.INDEX TERMS Usability, chatbots, experiments, family of experiments, systematic mapping study

show abstract

“…Software engineering experts designed the ideal class diagram before the experiment was carried out. To assess quality, we employed the following metrics [24]: Precision = TP / (TP + FP) TN (true negative): There are no true negatives in the model comparison; hence, the value is always 0.…”

Section: A Objectives Hypotheses and Variablesmentioning

confidence: 99%

Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings

Ren¹,

Pérez-Soler²,

Castro

et al. 2022

IEEE Access

View full text Add to dashboard Cite

After improving the SOCIO chatbot prototype model, we wanted to know how/if its usability has changed. An evidence-based empirical evaluation of the usability of SOCIO V1 (updated version) requires an extensive verification of the experimental results. A family of experiments is a method of verification whereby we can check if the experimental results are reproducible. Through comparison with the updated control tool Creately, we aimed to gain a better understanding of the usability of the collaborative modeling chatbot and how it could be improved based on experimental evidence of changes in terms of efficiency, effectiveness, satisfaction, and quality. A total of 87 students from three countries were recruited. We conducted a family of three experiments to compare the usability of SOCIO V1 and updated Creately in academic settings. Students appeared to be more satisfied with SOCIO V1, and SOCIO V1 scored better on completeness. There were no significant differences between the two tools regarding efficiency and quality. This study provides evidence on how to employ a family of experiments to improve chatbot usability and enrich knowledge on chatbot usability experimentation.INDEX TERMS Chatbot, usability, family of experiments.

show abstract

Engagement and Usability of Conversational Search – A Study of a Medical Resource Center Chatbot

Cited by 8 publications

References 36 publications

Using the SOCIO Chatbot for UML Modelling: A Family of Experiments

Using the SOCIO Chatbot for UML Modelling: A Family of Experiments

Experimentation for Chatbot Usability Evaluation: A Secondary Study

Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings

Contact Info

Product

Resources

About