Using Attributes of Survey Items to Predict Response Times May Benefit Survey Research

Schneider, Stefan; Jin, Haomiao; Orriens, Bart; Junghaenel, Doerte U.; Kapteyn, Arie; Meijer, Erik; Stone, Arthur A.

doi:10.1177/1525822x221100904

Cited by 3 publications

(3 citation statements)

References 30 publications

(34 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our second model was an expanded version of the location-scale model, based on the assumption that different components of intraindividual RT variability can be distinguished from each other that may relate to cognitive abilities in opposite ways. Specifically, survey items vary considerably in difficulty and the cognitive demands associated with them ( Schneider et al forthcoming ). In RT modeling, a concept analogous to item difficulty is the “time intensity” of an item, defined as “the amount of time an item tends to require” ( Kyllonen and Zu 2016, p. 14 ).…”

Section: Methodsmentioning

confidence: 99%

Inferring Cognitive Abilities from Response Times to Web-Administered Survey Items in a Population-Representative Sample

et al. 2022

View full text Add to dashboard Cite

Monitoring of cognitive abilities in large-scale survey research is receiving increasing attention. Conventional cognitive testing, however, is often impractical on a population level highlighting the need for alternative means of cognitive assessment. We evaluated whether response times (RTs) to online survey items could be useful to infer cognitive abilities. We analyzed >5 million survey item RTs from >6000 individuals administered over 6.5 years in an internet panel together with cognitive tests (numerical reasoning, verbal reasoning, task switching/inhibitory control). We derived measures of mean RT and intraindividual RT variability from a multilevel location-scale model as well as an expanded version that separated intraindividual RT variability into systematic RT adjustments (variation of RTs with item time intensities) and residual intraindividual RT variability (residual error in RTs). RT measures from the location-scale model showed weak associations with cognitive test scores. However, RT measures from the expanded model explained 22–26% of the variance in cognitive scores and had prospective associations with cognitive assessments over lag-periods of at least 6.5 years (mean RTs), 4.5 years (systematic RT adjustments) and 1 year (residual RT variability). Our findings suggest that RTs in online surveys may be useful for gaining information about cognitive abilities in large-scale survey research.

show abstract

Section: Methodsmentioning

confidence: 99%

Inferring Cognitive Abilities from Response Times to Web-Administered Survey Items in a Population-Representative Sample

et al. 2022

View full text Add to dashboard Cite

show abstract

“…This precision might also lead to the development of minimum thresholds for trial inclusion, which our data did not support, but which could be important for identifying inattentive responders in paid research. Indeed, our proposals for maximum thresholds might be further improved by modeling expected response times at the level of each questionnaire 30 . Second, we analyzed data from an online sample where participants completed modules on their own without supervision.…”

Section: Discussionmentioning

confidence: 99%

Digital questionnaire response time (DQRT): a ubiquitous and low-cost digital assay of cognitive status

Teckentrup,

Rosická,

Donegan

et al. 2024

Preprint

View full text Add to dashboard Cite

Digital survey tools have all but replaced paper and pen in the psychological sciences and consequently new forms of potentially useful research paradata are now routinely gathered. A particularly common byproduct of research are questionnaire timestamps, which some have suggested can be used as a measure of cognitive function. In this paper, we conducted a comprehensive validation of this measure, which we call the ‘digital questionnaire response time’ or ‘DQRT’. Using data from N=2979 users of a smartphone app, we examined how best to quantify DQRT, characterized its cognitive, clinical, and demographic correlates, established its psychometric properties, and highlighted substantial, but addressable, potential confounds inherent in the measure. We conclude that DQRT has important limitations, but on the whole can serve as a valid and reliable index of cognitive processing speed that can be gathered at unprecedented scale, unobtrusively, and repeatedly, during a variety of real-world digital behaviors.

show abstract

“…Following Jensen (2006), we refer to the term complexity as the information load involved in answering a self-report question. Information load cannot be assessed with a single attribute, and we coded 10 different characteristics of each question that are likely related to information load based on prior literature (Bais et al, 2019;Knäuper et al, 1997;Schneider, Jin et al, 2023a;Yan & Tourangeau, 2008) using four approaches, as described below.…”

Section: Indicators Of Question Complexitymentioning

confidence: 99%

Can you tell people’s cognitive ability level from their response patterns in questionnaires?

Schneider,

Hernandez,

Junghaenel

et al. 2024

Behav Res

Self Cite

View full text Add to dashboard Cite

Questionnaires are ever present in survey research. In this study, we examined whether an indirect indicator of general cognitive ability could be developed based on response patterns in questionnaires. We drew on two established phenomena characterizing connections between cognitive ability and people’s performance on basic cognitive tasks, and examined whether they apply to questionnaires responses. (1) The worst performance rule (WPR) states that people’s worst performance on multiple sequential tasks is more indicative of their cognitive ability than their average or best performance. (2) The task complexity hypothesis (TCH) suggests that relationships between cognitive ability and performance increase with task complexity. We conceptualized items of a questionnaire as a series of cognitively demanding tasks. A graded response model was used to estimate respondents’ performance for each item based on the difference between the observed and model-predicted response (“response error” scores). Analyzing data from 102 items (21 questionnaires) collected from a large-scale nationally representative sample of people aged 50+ years, we found robust associations of cognitive ability with a person’s largest but not with their smallest response error scores (supporting the WPR), and stronger associations of cognitive ability with response errors for more complex than for less complex questions (supporting the TCH). Results replicated across two independent samples and six assessment waves. A latent variable of response errors estimated for the most complex items correlated .50 with a latent cognitive ability factor, suggesting that response patterns can be utilized to extract a rough indicator of general cognitive ability in survey research.

show abstract

Using Attributes of Survey Items to Predict Response Times May Benefit Survey Research

Cited by 3 publications

References 30 publications

Inferring Cognitive Abilities from Response Times to Web-Administered Survey Items in a Population-Representative Sample

Inferring Cognitive Abilities from Response Times to Web-Administered Survey Items in a Population-Representative Sample

Digital questionnaire response time (DQRT): a ubiquitous and low-cost digital assay of cognitive status

Can you tell people’s cognitive ability level from their response patterns in questionnaires?

Contact Info

Product

Resources

About