Using a video game platform, we examined how vision-based decision making was affected by a concurrent, potentially conflicting auditory stimulus. Electroencephalographic responses showed that by 150 milliseconds of stimulus onset, the brain had detected the conflict between visual and auditory stimuli. Systematically reducing the intertrial interval (ITI), which subjects described as stressful, undermined decision making. Subjects’ arterial pulse variance decreased along with ITI, signaling increased parasympathetic influence on the heart. When successive trials required a shift in processing mode, short ITIs significantly boosted one trial’s influence on the next, suggesting that stress reduces cognitive flexibility. Finally, our study demonstrates the heart’s and the brain’s important influence on decision making.

Cross-modal interactions of auditory and visual temporal modulation were examined in a game-like experimental framework. Participants observed an audiovisual stimulus (an animated, sound-emitting fish) whose sound intensity and/or visual size oscillated sinusoidally at either 6 or 7 Hz. Participants made speeded judgments about the modulation rate in either the auditory or visual modality while doing their best to ignore information from the other modality. Modulation rate in the task-irrelevant modality matched the modulation rate in the task-relevant modality (congruent conditions), was at the other rate (incongruent conditions), or had no modulation (unmodulated conditions). Both performance accuracy and parameter estimates from drift-diffusion decision modeling indicated that (1) the presence of temporal modulation in both modalities, regardless of whether modulations were matched or mismatched in rate, resulted in audiovisual interactions; (2) congruence in audiovisual temporal modulation resulted in more reliable information processing; and (3) the effects of congruence appeared to be stronger when judging visual modulation rates (i.e., audition influencing vision), than when judging auditory modulation rates (i.e., vision influencing audition). The results demonstrate that audiovisual interactions from temporal modulations are bi-directional in nature, but with potential asymmetries in the size of the effect in each direction.

The human brain is excellent at integrating information from different sources across multiple sensory modalities. To examine one particularly important form of multisensory interaction, we manipulated the temporal correlation between visual and auditory stimuli in a first-person fisherman video game. Subjects saw rapidly swimming fish whose size oscillated, either at 6 or 8 Hz. Subjects categorized each fish according to its rate of size oscillation, while trying to ignore a concurrent broadband sound seemingly emitted by the fish. In three experiments, categorization was faster and more accurate when the rate at which a fish oscillated in size matched the rate at which the accompanying, task-irrelevant sound was amplitude modulated. Control conditions showed that the difference between responses to matched and mismatched audiovisual signals reflected a performance gain in the matched condition, rather than a cost from the mismatched condition. The performance advantage with matched audiovisual signals was remarkably robust over changes in task demands between experiments. Performance with matched or unmatched audiovisual signals improved over successive trials at about the same rate, emblematic of perceptual learning in which visual oscillation rate becomes more discriminable with experience. Finally, analysis at the level of individual subjects' performance pointed to differences in the rates at which subjects can extract information from audiovisual stimuli.

Boston's Museum of Science supports researchers whose projects advance science and provide educational opportunities to the Museum's visitors. For our project, 60 visitors to the Museum played “Fish Police!!,” a video game that examines audiovisual integration, including the ability to ignore irrelevant sensory information. Players, who ranged in age from 6 to 82 years, made speeded responses to computer-generated fish that swam rapidly across a tablet display. Responses were to be based solely on the rate (6 or 8 Hz) at which a fish's size modulated, sinusoidally growing and shrinking. Accompanying each fish was a task-irrelevant broadband sound, amplitude modulated at either 6 or 8 Hz. The rates of visual and auditory modulation were either Congruent (both 6 Hz or 8 Hz) or Incongruent (6 and 8 or 8 and 6 Hz). Despite being instructed to ignore the sound, players of all ages responded more accurately and faster when a fish's auditory and visual signatures were Congruent. In a controlled laboratory setting, a related task produced comparable results, demonstrating the robustness of the audiovisual interaction reported here. Some suggestions are made for conducting research in public settings.

BackgroundAn open, observational, three-arm clinical study aimed at investigating the efficacy of different neoadjuvant therapies (neoadjuvant immunotherapy with(out) chemotherapy, neoadjuvant chemotherapy, and neoadjuvant targeted therapy) in operable locally advanced non-small cell lung cancer (NSCLC) was conducted (NCT04197076). We report an interim analysis of 49 of 53 evaluable patients.MethodsThis study was conducted at Shanghai Chest Hospital and included eligible NSCLC patients who were 18 years old and had clinical stage IIB–IIIB disease. All 49 patients had surgical resection within 4–6 weeks after 2–3 cycles of neoadjuvant treatment consisting of immunotherapy (24 patients), chemotherapy (16 patients), and a targeted therapy (9 patients) regimen starting on the first day of each 21-day cycle. Pathologic complete response (pCR) was evaluated as the primary endpoint. Major pathological response (MPR) and tumor regression rate (TRR) were also evaluated.ResultsAn improved pathologic complete response was achieved in the neoadjuvant immunotherapy arm compared with the neoadjuvant chemotherapy arm and neoadjuvant targeted therapy arm [20.8% (5/24) vs. 6.3% (1/16) vs. 0.0% (0/9); P = 0.089, 95% CI 0.138–0.151]. More importantly, we found that the curative effect of the neoadjuvant immunotherapy arm in pCR+MPR was better than that of the neoadjuvant chemotherapy arm and neoadjuvant targeted therapy arm [45.8% (11/24) vs. 18.8% (3/16) vs. 0.0% (0/9); P = 0.006, 95% confidence interval, 0.008–0.012]. Different neoadjuvant therapies had a statistically significant effect on postoperative pathological tumor downstaging (P = 0.017).ConclusionsNeoadjuvant immunotherapy was associated with a trend toward better pCR than the neoadjuvant chemotherapy arm and neoadjuvant targeted therapy. Curative effect (pCR + MPR) was significantly better with neoadjuvant immunotherapy (P = 0.006, 95% confidence interval, 0.008–0.012).Clinical Trial Registrationhttps://clinicaltrials.gov/ct2/show/NCT04197076?recrs=a&cond=NCT04197076&draw=2&rank=1.