Investigating Lay Perceptions of Psychological Measures: A Registered Report

Abstract

In recent years, the reliability and validity of psychology measurement practices has been called into question, as part of an ongoing reappraisal of the robustness, reproducibility, and transparency of psychological research. While useful progress has been made, to date, the majority of discussions surrounding psychology’s measurement crisis have involved technical, quantitative investigations into the validity, reliability, and statistical robustness of psychological measures. This registered report offers a seldom-heard qualitative perspective to these ongoing debates, critically exploring members of the general publics’ (i.e., non-experts) lay perceptions of widely used measures in psychology. Using a combination of cognitive interviews and a think aloud study protocol, participants (n = 23) completed one of three popular psychology measures. Participants reflected on each of the measures, discussed the contents, and provided perceptions of what the measures are designed to test. Coding of the think aloud protocols concluded that participants across the measures regularly experienced issues in interpreting and responding to items. Thematic analysis of the cognitive interviews identified three dominant themes that each relate to lay perceptions of psychology measurements. These were: (1) participants’ grappling with attempting to ‘capture their multiple selves’ in the questionnaires, (2) participants perceived the questionnaire method as generally ‘missing nuance and richness’ and (3) exposing the ‘hidden labour of questionnaires’. These findings are discussed in the context of psychology’s measurement reform.

Different components of cognitive-behavioral therapy affect specific cognitive mechanisms

“…We note that, in general, participants found the internal-external dimension of choice options easier to parse than the global-specific dimension, which may explain the lack of robust effects on this measure. It is not now clear whether this is a limitation of our task materials or reflects a more general difficulty in understanding this aspect of attributional style, something that can be usefully explored in future work [e.g., (54)]. Further, it is an open question whether expression of these kinds of belief is a cause or consequence of low mood (55,56).…”

Different components of cognitive-behavioural therapy affect specific cognitive mechanisms

“…We note that, in general, participants found the internal-external dimension of choice options easier to parse than the global-specific dimension, which may explain the lack of robust effects on this measure. It is not currently clear whether this is a limitation of our task materials or reflects a more general difficulty in understanding this aspect of attributional style, something that can be usefully explored in future work (e.g., Mason et al 2023). Further, it is an open question whether expression of these kinds of belief is a cause or consequence of low mood (Cristea et al, 2015;Ezawa and Hollon, 2023).…”

Understanding Broader Community Perspectives on the Scientific Accuracy and Stigma of Personality Trait Labels

“…Assessing broader community perspectives on the meaning of diagnostic terminology is important from a dissemination and implementation perspective to assess the degree to which public perceptions of constructs align with definitions used by researchers and clinicians. More specifically, assessing community members' perspectives on the meaning of terminology can sharpen measurement approaches by providing insight into sources of information that individuals draw upon when providing information about themselves (e.g., how varying the title of a measure might influence responses; Mason et al, 2023), in addition to assessing the clarity of administration instructions and item wording. For example, assessment validity may be negatively impacted if individuals rely on different sources of information than researchers and clinicians intend when reporting on their personalities and mental health histories (Mason et al, 2023;Stanton et al, 2018).…”

“…Furthermore, it is possible that other terms not included here would be viewed more favorably than any of the terms included. Our assessment of community members' viewpoints was brief overall, and qualitative research assessing views of terminology in much more depth than was done here would be useful going forward (see Mason et al, 2023 for an example focused on personality assessment that used in-depth approaches with fewer participants).…”