Evaluation and Validation of a Conversational Agent Embodied in a Bookstore

Semeraro, Giovanni; Andersen, H. H.; Andersen, V.; Lops, Pasquale; Abbattista, Fabio

doi:10.1007/3-540-36572-9_28

Cited by 10 publications

(6 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Usability [40,56,[65][66][67][68][69] • User satisfaction [56,65,68,[70][71][72] • Agent credibility [70,73,74] The first two are common but difficult to measure attributes from the field of software engineering [59]. There are many more intangible and vague attributes mentioned in studies, including: "Fun to talk with" [38], "lovely, pleasant, black humorous" [70], "Intimacy, Benevolence" [37], "Comfort, Solidarity, Familiarity" [75] and "Trust, Uncertainty, Attractive" [66].…”

Section: Aspirational Subjective Measuresmentioning

confidence: 99%

“…There are many more intangible and vague attributes mentioned in studies, including: "Fun to talk with" [38], "lovely, pleasant, black humorous" [70], "Intimacy, Benevolence" [37], "Comfort, Solidarity, Familiarity" [75] and "Trust, Uncertainty, Attractive" [66].…”

Section: Aspirational Subjective Measuresmentioning

confidence: 99%

“…• Ease of use / Task ease [56,58,61,63,69,[76][77][78] • Ease of the user understanding the agent [56,58,63,66,76,77] • The agent's understanding of the user comprehension [56,65,66,77] • Various cognitive attributes related to comprehension and complexity [56,61,65,67,73,76] • Various attributes related to the reliability of the agent and the ease of correcting misunderstandings [65,76] • Various attributes concerning the user's expertise (of the domain or using the agent) [58,66,67,77] • The efficiency or effectiveness of the agent [56, 63, 65-67, 76, 78] • Various attributes about command and control of the conversation [65,73,76] [66, 67] • The pace of the interaction [76,77] • Whether the agent behaved as expected [56,58,61,64,65,77] • How natural the agent's behaviour seemed [61,73,…”

Section: Attempted Subjective Measuresmentioning

confidence: 99%

“…friendliness, enjoyment) [65,66] [63, 75, 76] • Various negative emotional attributes (e.g. boredom, fluster) [63,65,76] • Whether the user would use again [58,65,76,77] or prefer human service [56,61,76] There are also a substantial number of attributes which occur once or twice including "like further help" [65], "narrative skills" [70], "needs improvement" [76], "question answering capability" [70] and "how much willing to pay" [37].…”

Section: Attempted Subjective Measuresmentioning

confidence: 99%

See 3 more Smart Citations

Systems Engineering and Conversational Agents

O’Shea

Bandar

Crockett

2011

Intelligent Systems Reference Library

View full text Add to dashboard Cite

Abstract. This chapter describes Conversational Agents (CAs) in the context of Systems Engineering. A CA is a computer program which interacts with a user through natural language dialogue and provides some form of service. CA technology has two points of interest to systems engineers: the use of systems engineering techniques in CA research and the application of CAs in projects development. CAs offer the opportunity to automate more complex applications than are feasible with conventional web interfaces. Currently such applications require a human expert in the domain to mediate between the user and the application. The CA effectively replaces the human expert. This chapter reviews the current capabilities of various CA technologies, outlines a development methodology for systems engineering practitioners interested in developing real world applications and suggests a number of directions for systems engineers who wish to participate in CA research.Keywords: Conversational agent, systems engineering, dialogue, evaluation, methodology, semantic similarity, short text Section 1 IntroductionIn the early decades of computing ordinary people did not have any interaction with computers at all, any information that was needed for an application was entered by specialised clerks. This was followed by a period (largely driven by the internet) in which users could drive simple applications by entering simple facts and figures directly. Recently an explosion of internet use has demanded that ordinary people interact with complex applications, and the trend is for these to be of increasing complexity. Such applications would normally require a domain expert to interview the user to obtain the necessary information. Automating such processes represents a serious challenge to computing practitioners. This chapter describes a possible way forward, Conversational Agents (CAs). CA technology has two points of interest to systems engineers: the use of systems engineering techniques in CA research and the application of CAs in projects development. A CA is a computer program which interacts with a user through natural language dialogue and provides some form of service.

show abstract

Section: Aspirational Subjective Measuresmentioning

confidence: 99%

Section: Aspirational Subjective Measuresmentioning

confidence: 99%

Section: Attempted Subjective Measuresmentioning

confidence: 99%

Section: Attempted Subjective Measuresmentioning

confidence: 99%

See 2 more Smart Citations

Systems Engineering and Conversational Agents

O’Shea

Bandar

Crockett

2011

Intelligent Systems Reference Library

View full text Add to dashboard Cite

show abstract

“…Typically, subjective matters have involved a human user questionnaire. Semeraro et al [3] employ this technique for their bookstore chatbot. In the questionnaire, seven characteristics were appraised: impression, command, effectiveness, navigability, ability to learn, ability to aid, and comprehension.…”

Section: A Previous Approachesmentioning

confidence: 99%

Towards a method for evaluating naturalness in conversational dialog systems

Hung

Elvir

González

et al. 2009

2009 IEEE International Conference on Systems, Man and Cybernetics

View full text Add to dashboard Cite

Abstract-The evaluation of conversational dialog systems has always remained a controversial topic, as it is unclear on how to quantitatively describe how well a conversation agent performs, or how much better one is over another. Furthermore, one of the hurdles in this quandary remains the definition of naturalness, as in how well a dialog system can maintain a natural conversation flow. This paper surveys existing evaluation practices and provides its own methodology to determine the effectiveness and naturalness of a dialog system. A prototype evaluation methodology for the LifeLike virtual avatar project is presented.

show abstract

Context-Centric Speech-Based Human-Computer Interaction

Hung

González

2013

Int. J. Intell. Syst.

View full text Add to dashboard Cite

This paper describes research that addresses the problem of dialog management from a strong, context‐centric approach. We further present a quantitative method of measuring the importance of contextual cues when dealing with speech‐based human–computer interactions. It is generally accepted that using context in conjunction with a human input, such as spoken speech, enhances a machine's understanding of the user's intent as a means to pinpoint an adequate reaction. For this work, however, we present a context‐centric approach in which the use of context is the primary basis for understanding and not merely an auxiliary process. We employ an embodied conversation agent that facilitates the seamless engagement of a speech‐based information‐deployment entity by its human end user. This dialog manager emphasizes the use of context to drive its mixed‐initiative discourse model. A typical, modern automatic speech recognizer (ASR) was incorporated to handle the speech‐to‐text translations. As is the nature of these ASR systems, the recognition rate is consistently less than perfect, thus emphasizing the need for contextual assistance. The dialog system was encapsulated into a speech‐based embodied conversation agent platform for prototyping and testing purposes. Experiments were performed to evaluate the robustness of its performance, namely through measures of naturalness and usefulness, with respect to the emphasized use of context. The contribution of this work is to provide empirical evidence of the importance of conversational context in speech‐based human–computer interaction using a field‐tested context‐centric dialog manager.

show abstract

Evaluation and Validation of a Conversational Agent Embodied in a Bookstore

Cited by 10 publications

References 10 publications

Systems Engineering and Conversational Agents

Systems Engineering and Conversational Agents

Towards a method for evaluating naturalness in conversational dialog systems

Context-Centric Speech-Based Human-Computer Interaction

Contact Info

Product

Resources

About