How Do We Allocate Our Resources When Listening and Memorizing Speech in Noise? A Pupillometry Study

Abstract

Objectives: Actively following a conversation can be demanding and limited cognitive resources must be allocated to the processing of speech, retaining and encoding the perceived content, and preparing an answer. The aim of the present study was to disentangle the allocation of effort into the effort required for listening (listening effort) and the effort required for retention (memory effort) by means of pupil dilation. Design: Twenty-five normal-hearing German speaking participants underwent a sentence final word identification and recall test, while pupillometry was conducted. The participants’ task was to listen to a sentence in four-talker babble background noise and to repeat the final word afterward. At the end of a list of sentences, they were asked to recall as many of the final words as possible. Pupil dilation was recorded during different list lengths (three sentences versus six sentences) and varying memory load (recall versus no recall). Additionally, the effect of a noise reduction algorithm on performance, listening effort, and memory effort was evaluated. Results: We analyzed pupil dilation both before each sentence (sentence baseline) as well as the dilation in response to each sentence relative to the sentence baseline (sentence dilation). The pupillometry data indicated a steeper increase of sentence baseline under recall compared to no recall, suggesting higher memory effort due to memory processing. This increase in sentence baseline was most prominent toward the end of the longer lists, that is, during the second half of six sentences. Without a recall task, sentence baseline declined over the course of the list. Noise reduction appeared to have a significant influence on effort allocation for listening, which was reflected in generally decreased sentence dilation. Conclusion: Our results showed that recording pupil dilation in a speech identification and recall task provides valuable insights beyond behavioral performance. It is a suitable tool to disentangle the allocation of effort to listening versus memorizing speech.

Recall of Reverberant Speech in Quiet and Four-Talker Babble Noise

“…This finding is in line with our previous study performed on a group of NH listeners in quiet versus 4T conditions without any reverberation [20]. Consistent with the findings of Beatty and Kahneman (1966) [19] and Bönitz et al (2021) [21], the increasing trend in sentence baseline can be interpreted as increasing memory effort. In addition to the relative baseline correction used in this study, the latter study [21] analyzed the intercept and slope of sentence baseline and sentence dilation.…”

“…Consistent with the findings of Beatty and Kahneman (1966) [19] and Bönitz et al (2021) [21], the increasing trend in sentence baseline can be interpreted as increasing memory effort. In addition to the relative baseline correction used in this study, the latter study [21] analyzed the intercept and slope of sentence baseline and sentence dilation. The authors concluded that changes in baseline may reflect the participants' expectation about test difficulty even before the task was given.…”

“…We found that the sentence baseline increased consecutively in line with the increase in memory load (number of words to be recalled). Using the same correction method, a similar study by Bönitz et al (2021) [21] analyzed intercept and slope differences in sentence-baselines and dilations to compare three recall conditions with different list lengths (three or six words in a list) and two no-recall conditions. They observed that sentence baseline steeply increased under recall but declined under no-recall conditions.…”

The Effects of Task Difficulty Predictability and Noise Reduction on Recall Performance and Pupil Dilation Responses

“…Several measures have been used to capture the effects of adverse listening conditions and task demands on resource allocation ( Peelle 2018 ). In the current study, we combined the Sentence-Final Word Identification and Recall (SWIR) test ( Ng et al 2013 , 2015 ; Micula et al 2020 ) with pupillometry ( Wendt et al 2017 ; Ohlenforst et al 2018 ; Zhang et al 2021 ; Bönitz et al 2021 ), which are described individually in the following sections.…”

“… Bönitz et al (2021 ) have conducted a study combining pupillometry and a version of the SWIR test in German with lists of three and six sentences in a background noise of four-talker (4T) babble at high speech intelligibility levels. They included older participants with normal hearing.…”