Overview

Maier, Elisabeth; Mast, Marion; LuperFoy, Susann

doi:10.1007/3-540-63175-5_33

Cited by 1 publication

(1 citation statement)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The black box approach evaluates the performance of the system in terms of achieving its task, the cost of achieving the task in terms of time taken and number of turns, and measures the quality of the interaction, normally summarised by the term 'user satisfaction', which indicates whether the user " gets the information s/he wants, is s/he comfortable with the system, and gets the information within acceptable elapsed time, etc." (Maier et al 1996).…”

Section: Introductionmentioning

confidence: 99%

Different measurements metrics to evaluate a chatbot system

Shawar

Atwell

2007

Proceedings of the Workshop on Bridging the Gap Academic and Industrial Research in Dialog Technologies - NAACL-HLT '07

118

View full text Add to dashboard Cite

A chatbot is a software system, which can interact or "chat" with a human user in natural language such as English. For the annual Loebner Prize contest, rival chatbots have been assessed in terms of ability to fool a judge in a restricted chat session. We are investigating methods to train and adapt a chatbot to a specific user's language use or application, via a usersupplied training corpus. We advocate open-ended trials by real users, such as an example Afrikaans chatbot for Afrikaansspeaking researchers and students in South Africa. This is evaluated in terms of "glass box" dialogue efficiency metrics, and "black box" dialogue quality metrics and user satisfaction feedback. The other examples presented in this paper are the Qur'an and the FAQchat prototypes. Our general conclusion is that evaluation should be adapted to the application and to user needs.

show abstract

Section: Introductionmentioning

confidence: 99%