Most of the feedback received by operators of current robotteleoperation systems is graphical. When a large variety of robot data needs to be displayed however, this may lead to operator overload. The research presented in this paper focuses on offloading part of the feedback to other human senses, specifically to the sense of touch, to reduce the load due to the interface, and as a consequence, to increase the level of operator situation awareness. Graphical and vibro-tactile versions of feedback delivery for collision interfaces were evaluated in a search task using a virtual teleoperated robot. Parameters measured included task time, number of collisions between the robot and the environment, number of objects found and the quality of post-experiment reports through the use of sketch maps. Our results indicate that the combined use of both graphical and vibro-tactile feedback interfaces led to an increase in the quality of sketch maps, a possible indication of increased levels of operator situation awareness, but also a slight decrease in the number of robot collisions.KEYWORDS: virtual reality, robot teleoperation, multi-sensory interfaces, vibro-tactile feedback, collision proximity detection. INTRODUCTIONThe process of robot teleoperation may be divided into four primary activities: sensing the state of the robot and the remote environment, making sense of such state, deciding on the next action to be taken, and carrying out that action. Any of these steps may make use of automation. The human-robot interaction (HRI) cycle in Figure 1 happens indefinitely as the task is carried out. In the case of urban search-and-rescue (USAR), the main focus area of this paper, little automation is generally present, though the use of point navigation has become a common approach in robot teleoperation [22]USAR teleoperation is generally done through the use of ordinary input devices such as keyboard, mouse, and joystick. Most if not all of the information sensed from the robot is presented in a graphic display. During a mission, the operator uses this interface not only as a means to understand the state of the robot and its surrounding environment, but also as a tool to complete mission goals. Depending on how data is represented on screen, succeeding in both of these tasks may turn out to be very cognitively demanding. This increase in cognitive load may cause a decrease in operator situation awareness (SA) [12], and hence hinder the performance of the entire HRI system [10][24] [29].The research presented here aims at evaluating the impact on SA and performance when part of the data transmitted by the robot is displayed to the operator using senses other than vision. Specifically, the proposed interface uses a body-worn vibro-tactile display to provide feedback to the operator for collision proximity between the robot and the remote environment. In a four-way comparison, the use of vibro-tactile feedback is compared with the use of no feedback, the use of graphical feedback, and the use of both types of feed...

We present a novel approach for mixing real and computer-generated audio for augmented reality (AR) applications.Analogous to optical-see-through and video-see-through techniques in the visual domain, we present Hear-Through and Mic-Through audio AR. Hear-Through AR uses a bone-conduction headset to deliver computer-generated audio, while leaving the ear canals free to receive audio from the surrounding environment. Mic-Through AR allows audio signals captured from ear-worn microphones to be mixed with computer-generated audio in the computer, and delivered to the user over headphones. We present preliminary results from an empirical user study conducted to compare a bone-conduction device, headphones, and a speaker array. The results show that subjects achieved the best accuracy using an array of speakers physically located around the listener when stationary sounds were played, but that there was no difference in accuracy between the speaker array and the bone-conduction device for sounds that were moving, and that both devices outperformed standard headphones for moving sounds.

Multi-sensory displays provide information to users through multiple senses, not only through visuals. They can be designed for the purpose of creating a more-natural interface for users or reducing the cognitive load of a visual-only display. However, because multi-sensory displays are often application-specific, the general advantages of multi-sensory displays over visual-only displays are not yet well understood. Moreover, the optimal amount of information that can be perceived through multisensory displays without making them more cognitively demanding than visual-only displays is also not yet clear. Last, the effects of using redundant feedback across senses on multisensory displays have not been fully explored. To shed some light on these issues, this study evaluates the effects of increasing the amount of multi-sensory feedback on an interface, specifically in a virtual teleoperation context. While objective data showed that increasing the number of senses in the interface from two to three led to an improvement in performance, subjective feedback indicated that multi-sensory interfaces with redundant feedback may impose an extra cognitive burden on users.

Although multi-sensorial interfaces have been shown to improve user experience in different settings in Virtual Reality, these interfaces are not yet fully explored in urban search-and-rescue robot teleoperation. This paper presents a study on the performance effects of adding different types of vibro-tactile collision proximity feedback to a virtual robot's interface during a search task in a virtual environment. Results indicate that the addition of vibro-tactile feedback causes positive performance effects, especially for the intensity variation mode. Nevertheless, it also has a comfort impact for prolonged use.

Augmented reality (AR) is the mixing of computer-generated stimuli with real-world stimuli. In this paper, we present results from a controlled, empirical study comparing three ways of delivering spatialized audio for AR applications: a speaker array, headphones, and a bone-conduction headset. Analogous to optical-see-through AR in the visual domain, Hear-Through AR allows users to receive computer-generated audio using the bone-conduction headset, and real-world audio using their unoccluded ears. Our results show that subjects achieved the best accuracy using a speaker array physically located around the listener when stationary sounds were played, but that there was no difference in accuracy between the speaker array and the bone-conduction device for sounds that were moving, and that both devices outperformed standard headphones for moving sounds. Subjective comments by subjects following the experiment support this performance data. INTRODUCTIONAugmented reality (AR) is the mixing of computer-generated stimuli with real-world stimuli. While much work has been done for delivering mixed real-world (RW) and computer-generated (CG) stimuli in the visual domain, we focus here instead on the audio domain. Recently, we introduced two approaches for audio AR: Hear-Through AR and Mic-Through AR [8]. Our work on Hear-Through AR used either a speaker array or a bone-conduction headset (BCH) to deliver CG audio to a user, while RW sound was received through the unoccluded ear canals. Mic-Through AR captures RW sound using microphones mounted near the ears of the user, mixes it with CG sound in the computer, and delivers the resulting AR sound through standard headphones. In this paper, we present the first results from a formal, empirical study comparing subjects' sound-localization capabilities using both speaker-based and BCH-based Hear-Through AR, as well as Mic-Through AR. In order to gather baseline data, this study used only simple, well-controlled audio tones played at three frequencies. However, both static and moving tones were considered, while the head of the user was kept stationary. Lindeman & Noma [7] present a scheme for classifying AR techniques for all the human sensory modalities by where the mixing of CG and RW elements takes place. They underscore the need to correctly match the attributes of CG and RW stimuli so that the user can easily fuse the two, thereby improving the realism of the resulting mixed reality. Two main characteristics differentiate RW and CG audio. Real-world audio is typically of higher fidelity than CG audio. Also, computationally expensive preprocessing of CG audio is required in order to subject CG audio to similar environmental effects to match the RW environmental effects. Where the mixing of these elements takes place can have a significant impact on this computational cost.CG sound can be displayed using speakers placed within the real environment, allowing RW and CG sounds to mix in the environment before reaching the user. Alternatively, Mic-Through AR using two ...