Sherlock: Experimental Evaluation of a Conversational Agent for Mobile Information Tasks

Abstract

Controlled natural language (CNL) has great potential to support human-machine interaction (HMI) because it provides an information representation that is both human readable and machine processable. We investigated the effectiveness of a CNL-based conversational interface for HMI in a behavioral experiment called simple human experiment regarding locally observed collective knowledge (SHERLOCK). In SHERLOCK, individuals acted in groups to discover and report information to the machine using natural language (NL), which the machine then processed into CNL. The machine fused responses from different users to form a common operating picture, a dashboard showing the level of agreement for distinct information. To obtain information to add to this dashboard, users explored the real world in a simulated crowdsourced sensing scenario. This scenario represented a simplified controlled analog for tactical intelligence (i.e., direct intelligence of the environment), which is key for rapidly planning military, law enforcement, and emergency operations. Overall, despite close to zero training, 74% of the users inputted NL that was machine interpretable and addressed the assigned tasks. An experimental manipulation aimed to increase user-machine interaction, however, did not improve performance as hypothesized. Nevertheless, results indicate that the conversational interface may be effective in assisting humans with collection and fusion of information in a crowdsourcing context. Index Terms-Controlled natural language (CNL), conversational interface, human-computer collaboration (HCC), humanmachine interaction (HMI), tactical intelligence.

Usability of Chatbots: A Systematic Mapping Study

“…In [21], the authors deployed a digital pet avatar in the participants' home for 3 months to simulate real-life situations as much as possible. In [20], experiments were designed in a complex way, to simulate real-word situations. This is sometimes necessary, because if we were unable to use the chatbot effectively with a design as realistic as possible (but nonetheless simplified), it would be unlikely to be effective under more challenging conditions for the military, law enforcement and others in safety-critical realworld environments.…”

“…From the analysis of the papers, we found that questionnaires and interviews are most commonly used. Direct observation [20] Cognitive walkthrough [22] In most cases, two or more techniques are combined for the usability evaluation. Each of these methods has its own characteristics, and cannot fully meet all requirements of the usability test in isolation.…”

“…Hence, it is necessary to combine various methods. For example, in [20], direct observation and the System Usability Scale (SUS) questionnaire are jointly used. In [21], questionnaires and interviews are used together in every research phase.…”

“…In addition, less communication effort makes the chatbot easier to operate. In [20] the authors count the number of participants' cumulative assertions to measure the communication effort. Its steady increase demonstrates that users can use the chatbot efficiently in short time.…”

Automation bias with a conversational interface: User confirmation of misparsed information

“…We hypothesize that misparses result from automation bias and complacency [2], [3], a lack of situation awareness. Misparses were naturally occurring and a subset of data from simulated intelligence, surveillance, and reconnaissance (ISR) tasks in the real world collected using the SHERLOCK (Simple Human Experiments Regarding Locally Observed Collective Knowledge) platform [4]- [6].…”