Temporal binding of multimodal controls for dynamic map displays

Abstract

Dynamic map displays are visual interfaces that show the spatial positions of objects of interest (e.g., people, robots, vehicles), and can be updated with user commands as well as world changes, often in real time. Multimodal (speech and touch) controls were designed for a U.S. Army Research Laboratory dynamic map display to allow users to provide supervisory control of a simulated robotic swarm. This study characterized the effects of user performance (input difficulty, modality preference, and response to different levels of workload) on multimodal intercommand time (i.e., temporal binding), and explored how this might relate to the system's ability to bind or fuse user multimodal inputs into a unitary response. User performance was tested in a laboratory study using 6 male and 6 female volunteers with a mean age of 26 years. Results showed that 64% of all participants used speech commands first 100% of the time, while the remaining used touch commands first 100% of the time. Temporal binding between touch and voice commands was significantly shorter for touch-first than for speech-first commands, no matter what the level of workload. For both speech and touch commands, temporal binding was significantly shorter for both roads and swarm edges than for intersections. Results indicated that all of these factors can be significant in relating to a system's ability to bind multimodal inputs into a unitary response. Suggestions for future research are described.

Multimodal Interaction History and its use in Error Detection and Recovery

“…The overall consistencies are much lower (82.9%) and show a more divergent characteristic per user than reported in [7,9]. The same holds for another user classification based on the order of inputs [1,2,5]. None of these simple measures showed enough consistency, that promising error detection, based on them, could be expected.…”

“…In contrast to the findings of Oviatt et al, where typically simultaneous integration patterns were much more often observed, here, all of the inputs were sequential ones. Haas et al [2] conclude that this may be due to the fact, that their tasks involved much shorter (command-like) speech inputs than those of Oviatt et al…”

“…In [2] speech and single touch inputs are investigated for the task of placing different object types on a specific location of a map from a military scenario. The main measures, that may be relevant, are the input order (touch before speech, or speech before touch) and the time distances between the two modalities.…”

“…which representation of typical user behavior should be used, independent of the underlying task or modalities at hand? (2) In order to increase the robustness of a system, we must take a deeper look into the types of errors that can occur during multimodal input. And finally (3), how can an interaction history be actually used to detect, or even resolve, different error types in a multimodal input system?…”

“…In order to create a robust system, which has an error rate acceptable by the users, this factor could be of utmost importance. Studies in literature [1,2,5,[7][8][9], that have dealt with multimodal input behavior, show distinct interindividual differences, but, to our knowledge, were purely descriptive in nature. Little is known about which information, drawn from a user's interaction history, is actually relevant to achieve an increased robustness of a system as well as how such information can be applied in the first place.…”