Survey Item Validation

Abstract

In the social sciences, validity refers to the adequacy of a survey (or other mode of assessment) for its intended purpose. Validation refers to the activities undertaken during and after the construction of the survey to evaluate and improve validity. Item validation refers here to procedures for evaluating and improving respondents’ understanding of the questions and response options included in a survey. Verbal probing techniques such as cognitive interviews can be used to understand respondents’ response process, that is, what they are thinking as they answer the survey items. Although cognitive interviews can provide evidence for the validity of survey items, they are time-consuming and thus rarely used in practice. The Response Process Evaluation (RPE) method is a newly-developed technique that utilizes open-ended meta-surveys to rapidly collect evidence of validity across a population of interest, make quick revisions to items, and immediately test these revisions across new samples of respondents. Like cognitive interviews, the RPE method focuses on how participants interpret the item and select a response. The chapter demonstrates the process of validating one survey item taken from the Inventory of Non-Ordinary Experiences (INOE).

Mystical and Other Alterations in Sense of Self: An Expanded Framework for Studying Nonordinary Experiences

“…Because, in the case of the INOE, each item is its own construct, this is the primary evidence of validity. To validate the items, we are using online metasurveys in which we ask people to paraphrase the items, indicate how they would respond, and give actual or hypothetical examples of the experience (Wolf, Ihm, Maul, & Taves, in press). We code their responses as understood, sort of understood, or not understood on the basis of our definitional criteria.…”

Family Resemblance in Variations of Contemporary Religiosity and Spirituality: Findings from a Cross-Cultural Study

“…To achieve true multi-cultural validity involves item-by-item international, multilingual, and cross-cultural validation -before scales are made or any items put together (cf. Wolf et al, 2020). This is also a concern for both the development and translation of any Q-set to be used in a cross-cultural setting.…”

On the Design and Validation of Assessing Tools for Measuring the Impact of Programs Promoting STEM Vocations

“…The scale and the type of analysis are included in the third and fourth column, respectively. Wolf et al (2021). From the qualitative data, the answers given by all the participants were analyzed in parallel by each researcher to determine how the questionnaires worked in a real sample and to refine items if necessary.…”

“…The most common form of evaluation is the user survey (Robinson and Murray, 2019 ). When designed well and interpreted appropriately, self-report surveys can be used to gather useful data from large samples at relatively low-cost (Wolf et al, 2021 ). Note that informal education initiatives are usually constrained by low budgets and hence, sustainable implementations should be sought.…”