The roles of language processing in a spoken language interface.

Abstract

This paper provides an overview of the colloquium's discussion session on natural language understanding, which followed presentations by M. Bates [Bates, M. (1995) Proc This paper provides an overview of the natural language understanding session at the Colloquium on HumanMachine Communication by Voice held by the National Academy of Sciences (NAS). The aim of the paper is to review the role that language understanding plays in spoken language systems and to summarize the discussion that followed the two presentations by Bates (1) and Moore (2). A number of questions were raised during the discussion, including whether a single system could provide both understanding and constraint, what the future role of discourse should be, how to evaluate performance on interactive systems, and whether we are moving in the right direction toward realizing the goal of interactive human-machine communication.Background: The ARPA Spoken Language Program

Usefulness, localizability, humanness, and language-benefit: additional evaluation criteria for natural language dialogue systems

“…For example, glass box evaluation was applied on the ARPA Spoken Language system (Hirschman, 1995), and it shows that the error rate for sentence understanding was much lower than that for sentence recognition.…”

Different measurements metrics to evaluate a chatbot system

“…Different mechanisms are used to evaluate Spoken Dialogue Systems (SLDs), ranging from glass box evaluation that evaluates individual components, to black box evaluation that evaluates the system as a whole McTear (2002). For example, glass box evaluation was applied on the (Hirschman 1995) ARPA Spoken Language system, and it shows that the error rate for sentence understanding was much lower than that for sentence recognition. On the other hand black box evaluation evaluates the system as a whole based on user satisfaction and acceptance.…”