Vibrotactile Stimulation Based on the Fundamental Frequency Can Improve Melodic Contour Identification of Normal-Hearing Listeners With a 4-Channel Cochlear Implant Simulation

Verschuur

2021

Front. Neurosci.

Cochlear implants (CIs) have been remarkably successful at restoring speech perception for severely to profoundly deaf individuals. Despite their success, several limitations remain, particularly in CI users’ ability to understand speech in noisy environments, locate sound sources, and enjoy music. A new multimodal approach has been proposed that uses haptic stimulation to provide sound information that is poorly transmitted by the implant. This augmenting of the electrical CI signal with haptic stimulation (electro-haptic stimulation; EHS) has been shown to improve speech-in-noise performance and sound localization in CI users. There is also evidence that it could enhance music perception. We review the evidence of EHS enhancement of CI listening and discuss key areas where further research is required. These include understanding the neural basis of EHS enhancement, understanding the effectiveness of EHS across different clinical populations, and the optimization of signal-processing strategies. We also discuss the significant potential for a new generation of haptic neuroprosthetic devices to aid those who cannot access hearing-assistive technology, either because of biomedical or healthcare-access issues. While significant further research and development is required, we conclude that EHS represents a promising new approach that could, in the near future, offer a non-invasive, inexpensive means of substantially improving clinical outcomes for hearing-impaired individuals.

Section: Electro-haptic Stimulationmentioning

confidence: 99%

Electro-Haptic Stimulation: A New Approach for Improving Cochlear-Implant Listening

Verschuur

2021

Front. Neurosci.

“…However, this and other pitch extraction approaches for enhancing music perception using haptics (e.g., Luo and Hayes, 2019), are not designed to accommodate musical pieces with multiple simultaneous harmonic sounds. More advanced multipitch extraction methods will likely be required if they are to be effective across a range of musical pieces.…”

Section: Source Separationmentioning

confidence: 99%

“…Two further studies reported behavioral results for wearable devices. One wrist-worn device extracted the fundamental frequency, like the mosaicOne_B, but mapped it to changes in the frequency and amplitude of the haptic signal (which varied together), rather than spatial location ( Luo and Hayes, 2019 ). Critically, unlike for the mosaicOne_B, this meant that intensity information could not be delivered.…”

Section: Current Systems For Improving Music Perception Using Haptic Stimulationmentioning

confidence: 99%

“…Several studies have recently shown that haptic stimulation can enhance CI listening by allowing access to sound features that are poorly transferred through electrical CI stimulation (see Fletcher, 2020 ; Fletcher and Verschuur, 2021 ). This “electro-haptic stimulation” can substantially improve speech-in-noise performance ( Huang et al, 2017 ; Fletcher et al, 2018 , 2019 , 2020b ), sound localization ( Fletcher and Zgheib, 2020 ; Fletcher et al, 2020a ), and melody recognition ( Huang et al, 2019 ; Luo and Hayes, 2019 ), as well as discrimination of basic sound features such as pitch ( Fletcher et al, 2020c ). The impressive performance found in studies of haptic sound-localization and haptic enhancement of pitch discrimination suggests that it could also assist HA users ( Fletcher and Zgheib, 2020 ; Fletcher et al, 2020a , c ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Can Haptic Stimulation Enhance Music Perception in Hearing-Impaired Listeners?

2021

Front. Neurosci.

Cochlear implants (CIs) have been remarkably successful at restoring hearing in severely-to-profoundly hearing-impaired individuals. However, users often struggle to deconstruct complex auditory scenes with multiple simultaneous sounds, which can result in reduced music enjoyment and impaired speech understanding in background noise. Hearing aid users often have similar issues, though these are typically less acute. Several recent studies have shown that haptic stimulation can enhance CI listening by giving access to sound features that are poorly transmitted through the electrical CI signal. This “electro-haptic stimulation” improves melody recognition and pitch discrimination, as well as speech-in-noise performance and sound localization. The success of this approach suggests it could also enhance auditory perception in hearing-aid users and other hearing-impaired listeners. This review focuses on the use of haptic stimulation to enhance music perception in hearing-impaired listeners. Music is prevalent throughout everyday life, being critical to media such as film and video games, and often being central to events such as weddings and funerals. It represents the biggest challenge for signal processing, as it is typically an extremely complex acoustic signal, containing multiple simultaneous harmonic and inharmonic sounds. Signal-processing approaches developed for enhancing music perception could therefore have significant utility for other key issues faced by hearing-impaired listeners, such as understanding speech in noisy environments. This review first discusses the limits of music perception in hearing-impaired listeners and the limits of the tactile system. It then discusses the evidence around integration of audio and haptic stimulation in the brain. Next, the features, suitability, and success of current haptic devices for enhancing music perception are reviewed, as well as the signal-processing approaches that could be deployed in future haptic devices. Finally, the cutting-edge technologies that could be exploited for enhancing music perception with haptics are discussed. These include the latest micro motor and driver technology, low-power wireless technology, machine learning, big data, and cloud computing. New approaches for enhancing music perception in hearing-impaired listeners could substantially improve quality of life. Furthermore, effective haptic techniques for providing complex sound information could offer a non-invasive, affordable means for enhancing listening more broadly in hearing-impaired individuals.

“…More recently, in addition to substituting auditory input for haptic input, it has been shown that it is possible to augment auditory input with haptic input; three recent studies have shown that CI users' ability to recognise speech in background noise was enhanced when speech information was presented through haptic stimulation on the wrists 22,29 or fingertips 30 . Two other recent studies have shown that haptic stimulation can improve melody identification in CI users, using a single-channel haptic stimulation device strapped to the wrist 31 or fingertip 32 . In the current study, we evaluated the ability of our new mosaicOne_B device, which delivers pitch information through multi-channel haptic stimulation along the forearm, to provide accurate pitch information.…”

mentioning

confidence: 99%

Enhanced Pitch Discrimination for Cochlear Implant Users with a New Haptic Neuroprosthetic

Thini

Perry

2020

Sci Rep

the cochlear implant (ci) is the most widely used neuroprosthesis, recovering hearing for more than half a million severely-to-profoundly hearing-impaired people. However, CIs still have significant limitations, with users having severely impaired pitch perception. pitch is critical to speech understanding (particularly in noise), to separating different sounds in complex acoustic environments, and to music enjoyment. in recent decades, researchers have attempted to overcome shortcomings in cis by improving implant technology and surgical techniques, but with limited success. in the current study, we take a new approach of providing missing pitch information through haptic stimulation on the forearm, using our new mosaicOne_B device. The mosaicOne_B extracts pitch information in realtime and presents it via 12 motors that are arranged in ascending pitch along the forearm, with each motor representing a different pitch. In normal-hearing subjects listening to CI simulated audio, we showed that participants were able to discriminate pitch differences at a similar performance level to that achieved by normal-hearing listeners. furthermore, the device was shown to be highly robust to background noise. This enhanced pitch discrimination has the potential to significantly improve music perception, speech recognition, and speech prosody perception in ci users.