Capacity and tradeoffs in neural encoding of concurrent speech during Selective and Distributed Attention

Abstract

Speech comprehension is severely compromised when several people talk at once, due to limited perceptual and cognitive resources. Under some circumstances listeners can employ top-down attention to prioritize the processing of task-relevant speech. However, whether the system can effectively represent more than one speech input remains highly debated. Here we studied how task-relevance affects the neural representation of concurrent speakers under two extreme conditions: when only one speaker was task-relevant (Selective Attention), vs. when two speakers were equally relevant (Distributed Attention). Neural activity was measured using magnetoencephalography (MEG) and we analysed the speech-tracking responses to both speakers. Crucially, we explored different hypotheses as to how the brain may have represented the two speech streams, without making a-priori assumptions regarding participants' internal allocation of attention. Results indicate that neural tracking of concurrent speech did not fully mirror their instructed task-relevance. When Distributed Attention was required, we observed a tradeoff between the two speakers despite their equal task-relevance, akin to the top-down modulation observed during Selective Attention. This points to the system's inherent limitation to fully process two speech streams, and highlights the complex nature of attention, particularly for continuous speech.

An ecological investigation of the capacity to follow simultaneous speech and preferential detection of ones’ own name

“…Admittedly, here participants knew in advance which target-words to expect, which likely made this task easier for them. However, these results are in line with other studies showing that people can distribute their attention among two speech stimuli and accurately report content from both, even without prior expectations (Shinn-Cunningham and Ihlefeld, 2004; Shafiro and Gygi, 2007; Ihlefeld and Shinn-Cunningham, 2008; Gygi and Shafiro, 2014; Lambez et al, 2020; Kaufman and Golumbic, 2022). Given that distributing attention does not seem to bear considerable costs to performance, it seems likely that listeners might employ a distributed listening strategy in selective attention paradigms as well and monitor task- irrelevant speech even when not instructed to.…”

“…Moreover, adequate performance on this dual-task was not correlated with individual working-memory capacity (WMC), further supporting its relative low-cognitive-demand nature (Conway et al, 2001; Colflesh and Conway, 2007; Gygi and Shafiro, 2012; Naveh-Benjamin et al, 2014). At the same time, the neural speech tracking analysis shows that the Narrative Stream was represented more robustly than the Barista Stream, a pattern reminiscent of the enhanced speech-tracking of task-relevant speech in selective attention studies (Kerlin et al, 2010; Ding and Simon, 2012b, 2012a; Mesgarani and Chang, 2012; Power et al, 2012; Zion Golumbic et al, 2013; O’Sullivan et al, 2015; Fuglsang et al, 2017; Fiedler et al, 2019; Har-shai Yahav and Zion Golumbic, 2021; Kaufman and Golumbic, 2022). In discussing this result, we acknowledge the possibility that the lack of a reliable speech-tracking response to the Barista Stream may be due, at least in part, to the highly structured nature of this stimulus which contains substantial autocorrelation.…”

“…Given that distributing attention does not seem to bear considerable costs to performance, it seems likely that listeners might employ a distributed listening strategy in selective attention paradigms as well and monitor task- irrelevant speech even when not instructed to. This explanation would easily explain a wealth of findings regrading processing irrelevant speech, including the detection of ones’ own name (Conway et al, 2001; Dupoux et al, 2003; Rivenez et al, 2006, 2007, 2008; Best et al, 2010; Iyer et al, 2013; Röer et al, 2013; Naveh-Benjamin et al, 2014; Aydelott et al, 2015; Kidd and Arciuli, 2016; Röer et al, 2017; Har-shai Yahav and Zion Golumbic, 2021; Holtze et al, 2021; Kaufman and Golumbic, 2022). Considering the ‘ground-truth’ uncertainty regarding what listeners’ actually do in selective attention paradigms, and the relatively simple tasks they are asked to perform, this is not a possibility we can easily rule out.…”

“…Participants’ neural activity (electroencephalography; EEG) and physiological responses (galvanic skin response; GSR) were recorded throughout the experiment. Behavioral performance on the dual-task and analysis of the neural speech-tracking response to the Narrative and Barista Streams were used to assess the nature of potential ‘trade-offs’ in processing these concurrent speech stimuli which were both task-relevant, but in different ways (Kerlin et al, 2010; Ding and Simon, 2012a; Mesgarani and Chang, 2012; Power et al, 2012; Zion Golumbic et al, 2013; O’Sullivan et al, 2015; Fuglsang et al, 2017; Fiedler et al, 2019; Kaufman and Golumbic, 2022). We also studied the time-locked neural and physiological responses to target-words in the Barista Stream, complementing the behavioral data and providing insight into the potential ‘special status’ of detecting ones’ own name in a secondary speech stream.…”

Covert detection of own-name and semantic violations in task-irrelevant speech, in a realistic Virtual Café

“…Bearing these limitations in mind, the converging EEG and GSR results, showing responses for both the own name and semantic violation manipulations, provide strong indications the task-irrelevant barista-speech was monitored by the listeners, and that they were able to glean salient semantic information from it. This converges with the results of several recent studies, showing that listeners are actually capable of following the content of two simultaneous speakers, at least in contexts with moderate acoustic load (Agmon et al, 2021; Har-shai Yahav and Zion Golumbic, 2021; Kaufman and Zion Golumbic, 2022; Pinto et al, 2022).…”