Revisiting the DARPA communicator data using conversation analysis

Abstract

The state of the art in human computer conversation leaves something to be desired and, indeed, talking to a computer can be down-right annoying. This paper describes an approach to identifying "opportunities for improvement" in these systems by looking for abuse in the form of swear words. The premise is that humans swear at computers as a sanction and, as such, swear words represent a point of failure where the system did not behave as it should. Having identified where things went wrong, we can work backward through the transcripts and, using conversation analysis (CA) work out how things went wrong. Conversation analysis is a qualitative methodology and can appear quite alien -indeed unscientific -to those of us from a quantitative background. The paper starts with a description of Conversation analysis in its modern form, and then goes on to apply the methodology to transcripts of frustrated and annoyed users in the DARPA Communicator project. The conclusion is that there is at least one species of failure caused by the inability of the Communicator systems to handle mixed initiative at the discourse structure level. Along the way, I hope to demonstrate that there is an alternative future for computational linguistics that does not rely on larger and larger text corpora.

An Enactivist account of Mind Reading in Natural Language Understanding

“…The problem now is that those tasks have become definitive, and progress is now measured as being able to do those tasks slightly better. The real challenge is perhaps better characterised by the edge cases in the Communicator data [4], the Alexa Prize [5] and, dare I say it, the Loebner Prize [6].…”

Closing and closure in human-companion interactions: Analyzing video data from a field study

“…Practical experiences with today's conversational interactive systems show the deep gap between the utopian companion and today's clumsy attempts, see for example [2,3,4]. We are still far away from what, taken together, we could call "sociability", the skill, tendency or property of being sociable or social, of interacting well with others, involving elements of culture, situation, status, identity, task, communication, emotion, personality, and body, all at once and as an integrated whole.…”