Validity and reliability of four language mapping paradigms

Abstract

Language areas of the brain can be mapped in individual participants with functional MRI. We investigated the validity and reliability of four language mapping paradigms that may be appropriate for individuals with acquired aphasia: sentence completion, picture naming, naturalistic comprehension, and narrative comprehension. Five neurologically normal older adults were scanned on each of the four paradigms on four separate occasions. Validity was assessed in terms of whether activation patterns reflected the known typical organization of language regions, that is, lateralization to the left hemisphere, and involvement of the left inferior frontal gyrus and the left middle and/or superior temporal gyri. Reliability (test-retest reproducibility) was quantified in terms of the Dice coefficient of similarity, which measures overlap of activations across time points. We explored the impact of different absolute and relative voxelwise thresholds, a range of cluster size cutoffs, and limitation of analyses to a priori potential language regions. We found that the narrative comprehension and sentence completion paradigms offered the best balance of validity and reliability. However, even with optimal combinations of analysis parameters, there were many scans on which known features of typical language organization were not demonstrated, and test-retest reproducibility was only moderate for realistic parameter choices. These limitations in terms of validity and reliability may constitute significant limitations for many clinical or research applications that depend on identifying language regions in individual participants.

An adaptive semantic matching paradigm for reliable and valid language mapping in individuals with aphasia

“…This line of research critically depends on being able to identify brain regions involved in language processing in individual patients, and being able to determine with statistical rigor whether they change over time (Kiran et al, ; Meinzer et al, ; Wilson, Bautista, Yen, Lauderdale, & Eriksson, ). Language areas of the brain can be identified with functional magnetic resonance imaging (fMRI) using language mapping paradigms, which generally contrast conditions that involve language processing to conditions that do not (Binder, Swanson, Hammeke, & Sabsevitz, ).…”

“…This is referred to as test‐retest reproducibility. Research on neuroplasticity requires being able to distinguish genuine changes from scan‐to‐scan variability (Kiran et al, ; Meinzer et al, ; Wilson et al, ).…”

“…Acoustically matched control conditions, such as backwards speech, have typically been used. Probably due to the relatively unconstrained nature of both the language and the control conditions, the reliability of narrative comprehension paradigms has been empirically demonstrated to be moderate at best (Harrington, Buonocore, & Farias, ; Maldjian, Laurienti, Driskill, & Burdette, ; Wilson et al, ). The validity of narrative comprehension for mapping the language network is somewhat marginal.…”

Can neuroimaging help aphasia researchers? Addressing generalizability, variability, and interpretability

“…Specifically, the entire analysis pipeline—from data-driven generation of masks, through cross-validated fROI definition to group-level analyses of activations in these fROIs during the critical experimental task—can be performed using a publicly available toolbox (https://www.nitrc.org/projects/spm_ss). Moreover, localizer tasks for defining fROIs that are engaged in high-level language processing, including passive listening tasks that are suitable for PWAs, have been developed and thoroughly validated (Fedorenko et al, 2010; Scott et al, 2016; Stoppelman, Harpaz, & Ben‐Shachar, 2013; Tie et al, 2015; Wilson et al, 2016), and are available for download (https://evlab.mit.edu/funcloc/download-paradigms). In addition, these localizers are now available in 41 languages (https://evlab.mit.edu/alice).…”

“…However, as fMRI gained popularity among aphasia researchers, they have gradually recognized that adapting neuroimaging methodologies – originally developed for the study of neurologically intact populations – had to confront the same challenges faced by behavioral methods: establishing generalizability, taking into account inter-individual variability, and maintaining interpretability. These issues have prompted collaborative efforts to develop methodological guidelines for the use of fMRI in aphasia research (Crosson et al, 2007; Kiran et al, 2013; Meinzer et al, 2013; Price, Crinion, & Friston, 2006; Rapp, Caplan, Edwards, Visch-Brink, & Thompson, 2013; Veldsman, Cumming, & Brodtmann, 2015; Wilson, Bautista, Yen, Lauderdale, & Eriksson, 2016). Naturally, in striving to meet the three desiderata, a critical focus of these efforts has been to negotiate the relative benefits (and respective costs) of group and single case studies.…”