Conveying language through haptics

Abstract

In our daily lives, we rely heavily on our visual and auditory channels to receive information from others. In the case of impairment, or when large amounts of information are already transmitted visually or aurally, alternative methods of communication are needed. A haptic language offers the potential to provide information to a user when visual and auditory channels are unavailable. Previously created haptic languages include deconstructing acoustic signals into features and displaying them through a haptic device, and haptic adaptations of Braille or Morse code; however, these approaches are unintuitive, slow at presenting language, or require a large surface area. We propose using a multi-sensory haptic device called MISSIVE, which can be worn on the upper arm and is capable of producing brief cues, sufficient in quantity to encode the full English phoneme set. We evaluated our approach by teaching subjects a subset of 23 phonemes, and demonstrated an 86% accuracy in a 50 word identification task after 100 minutes of training.

Identification of Words and Phrases Through a Phonemic-Based Haptic Display

“…Thus, it is not known whether these same rates of phonemic presentation would allow for comparable word-identification performance after an increase to the full set of 39 English phonemes. Dunkelberger et al (2018) [34] developed tactile codes for 23 English phonemes and used a set of 150 words from which stimuli were drawn for training and testing. Unlike in the current study where the inter-phoneme interval was set by the experimenter, a self-paced rate of presentation was used between the phonemes in a given word, resulting in an average phoneme presentation rate of 3.5 sec/phoneme.…”

“…Post-training word-identification scores averaged 87% correct, similar to the word scores with IPI in the range of 75 to 300 ms in Study 1 here. However, the rates at which word stimuli were presented here were roughly eight times faster than those of Dunkelberger et al (2018) [34]. In the work of Fontana de Vargas et al [35], 24 tactile phonemes were encoded with a phoneme presentation rate of 3.1 sec/phoneme, similar to that of Reference [34].…”

“…Mean performance was 86% for the group with fixed-schedule training and 72% correct for self-guided training, although significantly more participants from the former group were able to achieve performance >90% correct. Dunkelberger et al (2018) [34] developed tactile codes for 23 English phonemes using a display attached to the upper forearm that consisted of four tactors, a radial squeeze band, and a lateral skin stretch rocker. After 100 min of training on identification of the 23 individual phonemes and of a set of 150 words composed of these phonemes, participants were tested using a subset of 50 words selected from this vocabulary.…”

“…Recent work has employed a phonemic-based strategy for a tactile display of speech, under the assumption that phonemic strings can be derived from ASR at the front end of the system [25][26][27][28][29][30][31][32][33][34][35]. This approach allows for the development of tactile displays that are no longer tied to displays of the acoustic speech spectrum, but instead allow for the design of displays that exploit the information-bearing properties of the tactual sense.…”

ISWC 2018: Towards Smarter Wearable Technologies That Blend With Our Bodies

“…Two research groups presented work on communicating language through haptic stimuli. Dunkelberger et al 3 describe a multiactuator haptic device that can be worn on the upper arm and is capable of producing brief cues to encode English phonemes. Luzhnica et al train the skill of comprehending text from vibrotactile patterns using passive haptic learning, a method that teaches manual skills while requiring little to no active attention by the user.…”