Predicting Decisions in Human Social Interactions Using Real-Time fMRI and Pattern Classification

Abstract

Negotiation and trade typically require a mutual interaction while simultaneously resting in uncertainty which decision the partner ultimately will make at the end of the process. Assessing already during the negotiation in which direction one's counterpart tends would provide a tremendous advantage. Recently, neuroimaging techniques combined with multivariate pattern classification of the acquired data have made it possible to discriminate subjective states of mind on the basis of their neuronal activation signature. However, to enable an online-assessment of the participant's mind state both approaches need to be extended to a real-time technique. By combining real-time functional magnetic resonance imaging (fMRI) and online pattern classification techniques, we show that it is possible to predict human behavior during social interaction before the interacting partner communicates a specific decision. Average accuracy reached approximately 70% when we predicted online the decisions of volunteers playing the ultimatum game, a well-known paradigm in economic game theory. Our results demonstrate the successful online analysis of complex emotional and cognitive states using real-time fMRI, which will enable a major breakthrough for social fMRI by providing information about mental states of partners already during the mutual interaction. Interestingly, an additional whole brain classification across subjects confirmed the online results: anterior insula, ventral striatum, and lateral orbitofrontal cortex, known to act in emotional self-regulation and reward processing for adjustment of behavior, appeared to be strong determinants of later overt behavior in the ultimatum game. Using whole brain classification we were also able to discriminate between brain processes related to subjective emotional and motivational states and brain processes related to the evaluation of objective financial incentives.

Online decoding of object‐based attention using real‐time fMRI

“…In light of this new view, optimal decoding of faces and places from these regions call for a multivariate decoding approach that can detect these overlapping and distributed neural patterns. Therefore, in this study, we used whole‐brain data to train a classifier to predict the mental state of a subject as this approach does not rely on any prior assumptions about functional localization (Laconte et al ., ; Anderson et al ., ; Hollmann et al ., ; Lee et al ., ; Xi et al ., ; DeBettencourt et al ., ). Moreover, the whole‐brain decoder is highly suited for real‐time fMRI because it automatically identifies sparse and distributed patterns of activity that are representation‐specific.…”

Neural Networks and Neurofeedback in Parkinson’s Disease

“…These trained pattern classifiers can then predict subsequent manifest cognitive states by reading the evoked neural activation patterns (represented by a spatially distributed set of activated voxels in the context of fMRI), based on the experience gained during training (for review, see Haynes & Rees, 2006). Pattern classifiers have been applied in real-time fMRI paradigms (Hollmann et al, 2011;LaConte, Peltier, & Hu, 2007;Sitaram et al, 2011) and have been shown to improve performance in perceptual discrimination tasks through neurofeedback training (Shibata, Watanabe, Sasaki, & Kawato, 2011). MVPA could potentially be utilized to overcome the limitations of univariate real-time fMRI methods in the context of PD, by avoiding an a priori approach and by closely modeling the complex network dynamics that underpin this pathology.…”

Research Methods in Social and Affective Neuroscience

“…A recent study used this technique to demonstrate a 75% accuracy rate on trained words, and about a 60% accuracy on untrained but semantically related words (Mitchell, Shinkareva, Carlson, Chang, Malave, Mason et al, 2008), and another used it to communicate with minimally conscious patients using a yes/no response (Monti, Vanhaudenhuyse, Coleman, Boly, Pickard, Tshibanda et al, 2010). One group of researchers combined pattern classification with real-time analysis to successfully predict decisions in real time during an ultimatum game (Hollmann, Rieger, Baecke, Lützkendorf, Müller, Adolf et al, 2011). As elegant as they are, note that all of these studies either require many exemplars from the same individual or make prediction from one individual to a large group of others (i.e., on average), but none can yield the content of a given thought without at least some prior information.…”