How and Why Affective and Reactive Virtual Agents Will Bring New Insights on Social Cognitive Disorders in Schizophrenia? An Illustration with a Virtual Card Game Paradigm

Abstract

In recent decades, many studies have shown that schizophrenia is associated with severe social cognitive impairments affecting key components, such as the recognition of emotions, theory of mind, attributional style, and metacognition. Most studies investigated each construct separately, precluding analysis of the interactive and immersive nature of real-life situation. Specialized batteries of tests are under investigation to assess social cognition, which is thought now as a link between neurocognitive disorders and impaired functioning. However, this link accounts for a limited part of the variance of real-life functioning. To fill this gap, advances in virtual reality and affective computing have made it possible to carry out experimental investigations of naturalistic social cognition, in controlled conditions, with good reproducibility. This approach is illustrated with the description of a new paradigm based on an original virtual card game in which subjects interpret emotional displays from a female virtual agent, and decipher her helping intentions. Independent variables concerning emotional expression in terms of valence and intensity were manipulated. We show how several useful dependant variables, ranging from classic experimental psychology data to metacognition or subjective experiences records, may be extracted from a single experiment. Methodological issues about the immersion into a simulated intersubjective situation are considered. The example of this new flexible experimental setting, with regards to the many constructs recognized in social neurosciences, constitutes a rationale for focusing on this potential intermediate link between standardized tests and real-life functioning, and also for using it as an innovative media for cognitive remediation.

Virtual Avatar for Emotion Recognition in Patients with Schizophrenia: A Pilot Study

“…The use of virtual agents constitutes a potential intermediate link between standardized tests and real-life functioning, and also offers the possibility of being used as an innovative medium for cognitive remediation (Oker et al, 2015 ). As such, the objective of the present study is to explore the possibilities of using a realistic-looking avatar previously introduced in Marcos et al ( 2010 ) for the assessment of emotion recognition deficits in patients who suffer schizophrenia.…”

The Factor Structure of Social Cognition in Schizophrenia: A Focus on Replication With Confirmatory Factor Analysis and Machine Learning

“…Alternatively, tasks developed for other clinical populations, such as autism spectrum and neurological disorders (Caruana et al, 2018;Henry, Von Hippel, Molenberghs, Lee, & Sachdev, 2016;Kandalaft, Didehbani, Krawczyk, Allen, & Chapman, 2013), might be fruitfully adapted to schizophrenia research. There are also interesting new paradigms using emerging technologies, such as virtual reality, that show promise for use in schizophrenia research (Caruana, Seymour, Brock, & Langdon, 2019;Oker et al, 2015).…”

Modeling Emotional Valence Integration From Voice and Touch

“…Using two channels simultaneously changed the perception of emotions compared to a unimodal channel of communication. Obviously, in multimodal social interactions the interpretation of affect is modulated by the context in which the stimulations (whether visual, audio, and/or tactile) take place, and the familiarity or relationship with person, or artificial agent, may it be a virtual human ( Oker et al, 2015 ) or humanoid robot ( Artstein et al, 2016 ). Therefore, future research is needed to evaluate to what extent context might override subtle differences in stimulus qualities.…”