CoWME

Abstract

Evaluating human machine interaction in the case of multimodal systems is often a difficult task involving the monitoring of multiple sources, data fusion and results interpretation. While subtasks are highly dependent on the specific goal of the application and on the available interaction modalities, it is possible to formalize this workflow into a standard process and to consider a generic measure to estimate the ease of use of a specific application. In this work, we present CoWME, a modular software architecture describing multimodal human machine interaction evaluation, from data collection to final evaluation, in a formal way, in terms of cognitive workload. Communication protocols between modules are described in XML while data fusion is delegated to a configurable rule engine. An interface module is introduced between the monitoring modules and the rule engine to collect and summarize data streams for cognitive workload evaluation. We present a deployment example showing how this architecture is deployed by monitoring an interactive session with an Android application taking into account stressed speech detection, mydriasis and touch analysis.

On data-driven systems analyzing, supporting and enhancing users’ interaction and experience

“…), first of all, it should be noted that the software architectures presented are intended mostly to support interaction, not to analyze it. The only paper that clearly presented a software architecture that was fully intended to support interaction analysis was [133]. That paper presented a software architecture that supported and analyzed interaction with a multimodal system.…”

“…Sciences, Education, Medicine, Business), the selected papers proposed software architectures to solve HCI/HMI issues related to different application domains. As illustrated in Figure 5, the software architectures were applied to domains such as aeronautics [129] (2.56%), BCI systems [134,152] (5.12%), cognitive systems [144] (2.56%), collaboration systems [154] (2.56%), context-aware systems [136,150,159] (7.7%), eyewear computing [142] (2.56%), healthcare systems [131,135,146,148] (10.25%), industrial machines [130,132] (5.12%), mobile systems [143,149,164] (7.7%), multimodal systems [128,133,137,140,155,156,163] (17.95%), robotics [138,139,157,158,160,161] (15.38%), smart environments [147] (2.56%), software engineering [17,162] (8.16%), ubiquitous environments [145] (2.56%), vehicle systems [141] (2.56%) or eLearning systems [127] (2.56%). As for question MQ5 (which kinds of devices have been involved in software architectures supporting or analyzing HCI / HMI processes?…”

“…), the papers showed different kinds of support provided by the software architectures. As illustrated in Figure 6, the software architectures included support to varying HCI/HMI processes, including the following: 7 papers [127,128,135,140,147,155,157] were related to the multimodal interaction support provided by software architectures (17.95% of the papers), in 6 papers [138,139,144,158,160,161] the software architectures supported human-robot interaction (15.38%), in 5 papers [17,149,151,153,164] these aimed to improve usability (12.82%), in another 5 [130,131,134,141,152] they supported human-machine interaction (12.82%), in 4 papers [136,145,148,159] software architectures were related to articulate context-awareness (10.26%), in 3 [137,150,163] they defined software agents to adapt software architectures to different contexts (7.69%), in 1 paper [129] the software architecture supported the proper connection of software and hardware components (2.56%), in [146] it defined interaction models to use with 3D contents (2.56%), in [132] it supported the evaluation of touch interaction (2.56%), in [142] it supported eye interaction with a system (2.56%), in [162] it aimed to support the integration of software requirements and HCI (2.56%), [133] aimed to analyze multimodal interactions (2.56%), in…”

“…In general, the authors reported a positive evaluation of a system and prototype developed using their approach and described how this kind of requirements techniques could be applied jointly to other design methodologies and methods such as storyboards, scenarios, and prototypes to achieve better products and receive positive feedback from the users. To finish the discussion of this group of papers, the last three papers were related to multimodal interaction: the conference paper [133], by Calandra, Caso, Cutugno, Origlia, and Rossi presented CoWME, a framework and software architecture designed to evaluate cognitive workload during multimodal interaction using different systems. This software architecture, in words of the authors, "takes as input an arbitrary number of input streams containing data reports about the cognitive workload estimated by dedicated modules, each concentrating on specific modalities.…”