Think-aloud analysis of commonly used screening instruments for Internet use disorders: The CIUS, the IGDT-10, and the BSMAS

Abstract

Background Despite the constant publication of new screening instruments for Internet use disorders (IUD), little is known about their content validity. This study aimed to identify potential mismatches between the items' intention and young adults' interpretation of these items when answering three screening instruments that are commonly used in research and clinical settings: The Compulsive Internet Use Scale (CIUS), the 10 Item-Internet Gaming Disorder Test (IGDT-10), and the Bergen Social Media Addiction Scale (BSMAS). Methods In total, 30 vocational students (50% female, age = 21.3; SD = 2.1) took part in individual think-aloud interviews. All participants were asked to report their thoughts while completing the CIUS. In addition, participants who reported online games (OG) as their main Internet activity (n = 11) answered the IGDT-10. Participants who reported social networks (SN) as their main Internet activity (n = 18) answered the BSMAS. One participant used OG and SN equally and completed both screening instruments. All interviews were audio-recorded, transcribed, and content-analysed. Results Overall, four potential sources for errors were identified: (1) High scorings were often not congruent with the underlying diagnostic criteria. In particular, such discrepancies were found in items aimed to assess dysfunctional emotional regulation strategies and cognitive involvement. (2) Some participants were uncertain which time frame or Internet activity should be considered in their answers. (3) Long and complex items led to the building of mean values or the choice of the middle answer category. (4) Some wordings were perceived to be outdated and difficult to understand. Discussion These findings might help to provide recommendations on how to improve screening instruments for IUD. Most important, items should more clearly distinguish between Internet use as a “normal” leisure activity and Internet use that leads to functional impairments in daily life.

Assessment of Criteria for Specific Internet-use Disorders (ACSID-11): Introduction of a new screening instrument capturing ICD-11 criteria for gaming disorder and other potential Internet-use disorders

“…The items were derived in multiple discussions and consensus meetings based on ICD-11 criteria for disorders due to addictive behaviors, as they are described for gaming and gambling, assuming a multifactorial structure. Findings of a Talk-Aloud Analysis were used to optimize content validity and comprehensibility of the items ( Schmidt et al, submitted ).…”

Investigating lay perceptions of psychological measures: A registered report

“…Think aloud protocols (TAPs) involve participants continuously verbalising their thoughts while completing each item in a questionnaire in such a manner that they do not disrupt their own thought process (Ericsson & Simon, 1998;Gilhooly & Green, 1996). Think aloud has previously been used to uncover a wide array of issues with self-report questionnaires; from participants misunderstanding terminology used in some self-report health questionnaires (e.g., Willis et al, 1991), outdated and complicated question word ing (e.g., Schmidt, Brandt, et al, 2022), specific difficulties with various sub-constructs (e.g., van Oort et al, 2011), and how question wording can bias choices towards middle values (e.g., Darker & French, 2009). Research demonstrates that participants tend to experience four major processes when answering items on a questionnaire: question comprehension, retrieval of information relevant to the question, decision making, and generation of their response (Tourangeau et al, 2000).…”

“…Our core research question was: "How do non-experts understand the contents, wording, and structure of popular psychology measures? ", which was approached using the think aloud method, followed by cognitive interviewing, with both of these methods being used in previous research (e.g., Belzer et al, 2013;Darker & French, 2009;and Schmidt, Brandt, et al, 2022). The present study is important as it is, to the best of our knowledge, the first to make use of Wolcott and Lobczowski's (2021) notion of the utility of TAPs and CIs for reappraising psychological measurement.…”

Daily motivational underpinnings and outcomes of adolescent video gaming

“…), each measuring whether the participants experienced a given Gaming Disorder symptom during the last year. In accordance with the insights obtained in a previously conducted qualitative study and in another study as well (Schmidt et al, 2022) "During last year…" was added at the beginning of every item for the participants to be aware that the items refer to their experiences during last year. Items were answered on a 5-point scale, from 1-never to 5-very often.…”