A re-defined and generalized percent-overlap-of-activation measure for studies of fMRI reproducibility and its use in identifying outlier activation maps

Abstract

Functional Magnetic Resonance Imaging (fMRI) is a popular noninvasive modality to investigate activation in the human brain. The end result of most fMRI experiments is an activation map corresponding to the given paradigm. These maps can vary greatly from one study to the next, so quantifying the reliability of identified activation over several fMRI studies is important. The percent overlap of activation (Rombouts et al., 1998 andMachielsen et al., 2000) is a global reliability measure between activation maps drawn from any two fMRI studies. A slightly modified but more intuitive measure is provided by the Jaccard (1901) coefficient of similarity, whose use we study in this paper. A generalization of these measures is also proposed to comprehensively summarize the reliability of multiple fMRI studies. Finally, a testing mechanism to flag potentially anomalous studies is developed. The methodology is illustrated on studies involving left-and righthand motor task paradigms performed by a right-hand dominant male subject several times over a period of two months, with excellent results. KeywordsEigenvalues, Reliability, Intra-class correlation coefficient, Outlier detection, Percent overlap, Principal components, Finger-thumb opposition experiment, Summarized multiple Jaccard similarity coefficient, Dice coefficient, Perron-Frobenius theorem Disciplines Statistics and ProbabilityComments NOTICE:this is the author's version of a work that was accepted for publication in NeuroImage. Changes resulting from the publishing process, such as peer review, editing, corrections structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in NeuroImage, [v.50, iss.1,(2010) AbstractFunctional Magnetic Resonance Imaging (fMRI) is a popular non-invasive modality to investigate activation in the human brain. The end result of most fMRI experiments is an activation map corresponding to the given paradigm. These maps can vary greatly from one study to the next, so quantifying the reliability of identified activation over several fMRI studies is important. The percent overlap of activation (Rombouts et al., 1998;Machielsen et al., 2000) is a global reliability measure between activation maps drawn from any two fMRI studies. A slightly modified but more intuitive measure is provided by the Jaccard (1901) coefficient of similarity, whose use we study in this paper. A generalization of these measures is also proposed to comprehensively summarize the reliability of multiple fMRI studies. Finally, a testing mechanism to flag potentially anomalous studies is developed. The methodology is illustrated on studies involving left-and right-hand motor task paradigms performed by a right-hand dominant male subject several times over a period of two months, with excellent results.

Motor map reliability and aging: a TMS/fMRI study

“…CoG shift was calculated as the average coordinate distance oVset, measured by the two dimensional Euclidean distances between COG locations from mapping sessions 1-2, mapping sessions 2-3, and mapping sessions 1-3 (Wolf et al 2004). Intersession TMS map reliability was assessed in two ways: (a) with the intraclass correlation coeYcient (ICC) and (b) with the modiWed Jaccard CoeYcient for spatial reliability (see Maitra 2010) of the three mapping sessions of each participant.…”

“…ICC values above 0.50 were considered as having moderate intersession reliability, while values over 0.75 were considered highly reliable (Portney and Watkins 2000). Second, we used a minor modiWcation of the Jaccard CoeYcient (Maitra 2010) to characterize between session changes in map metrics across groups. This metric is used to measure spatial reliability for test-retest examinations by indexing the number of areas (stimulation sites in TMS or voxels in fMRI) that are "active" in a given map.…”

“…This metric is used to measure spatial reliability for test-retest examinations by indexing the number of areas (stimulation sites in TMS or voxels in fMRI) that are "active" in a given map. The modiWed Jaccard CoeYcient, as implemented, is based on the original Jaccard similarity index (Jaccard 1901) but speciWed by the formula: J = M11/(M01 + M10 ¡ M11); where M11 represents the number of coactive sites/voxels in both comparisons sessions and both M01 and M10 represent sites/voxels active in only one of the two comparisons sessions, respectively (Maitra 2009(Maitra , 2010. Based on a hypothesis of greater reliability of motor maps in younger adults, we performed directional hypothesis tests on the Jaccard CoeYcient of the motor maps across both TMS and fMRI for all session comparisons (i.e., Session 1 vs.…”

An Iterative Jackknife Approach for Assessing Reliability and Power of fMRI Group Analyses

“…Figure 3). It is also not aimed at evaluating the reproducibility between repeated sessions [23], and is not used to resolve interdependence [16]. Also, in contrast to the approach taken by McNamee & Lazar [24], it is specifically not aimed at detecting outliers.…”

“…is “reliable”). This constitutes a group percent overlap map (gPOM), similar to approaches used previously [16], [23], [24].…”