The Response Process Evaluation Method

Abstract

Pretesting survey items for interpretability and relevance is a commonly recommended practice in the social sciences. The goal is to construct items that are understood as intended by the population of interest and test if participants use the expected cognitive processes when responding to a survey item. Such evidence forms the basis for a critical source of validity evidence known as the response process, which is often neglected in favor of quantitative methods. This may be because existing methods of investigating item comprehension, such as cognitive interviewing and web probing, lack clear guidelines for retesting revised items and documenting improvements, and can be difficult to implement in large samples. To remedy this, we introduce the Response Process Evaluation (RPE) method. The RPE method is a standardized framework for pretesting multiple versions of survey items and generating individual item validation reports. This iterative, evidence-based approach to item development relies on feedback from the population of interest to quantify and qualify improvements in item interpretability across a large sample. The result is a set of item validation reports that detail the intended interpretation and use of each item, the population it was validated on, the percent of participants that interpreted the item as intended, examples of participant interpretations, and any common misinterpretations to be cautious of. Researchers may find that they have more confidence in the inferences drawn from survey data after engaging in rigorous item pretesting.

The Inventory of Nonordinary Experiences (INOE): Evidence of validity in the United States and India

“…In other words, the design is premised on our ability to strip experiences, described phenomenologically, of culture-laden concepts, while at the same time ensuring that items remain comprehensible. To test these two assumptions of the INOE we used the RPE method, a novel technique for item-level validation [ 14 , 15 ].…”

“…Here, we present a new survey instrument–The Inventory of Nonordinary Experiences (INOE)–that queries a wide range of features and appraisals, allowing researchers to make cross-cultural comparisons between English-speakers in the US and Hindi-speakers in India. To ensure that the INOE survey items are understood as intended cross culturally, we present evidence of item level validity using the newly developed Response Process Evaluation (RPE) method [ 14 , 15 ].…”

“…Then, in the second step, we used a newly-developed item validation process–the RPE method–to test whether a proposed etic was actually understood as intended in the cultures to be compared and thus could be considered a derived etic . The RPE method, which turns cognitive interviews into open-ended meta-surveys through the use of web probes, allowed us to assess whether respondents understood items as intended in both cultural contexts and, if not, to iteratively revise and re-test them [ 14 , 15 , 26 ]. In contrast to Berry’s approach in which researchers identify the common feature based on their understanding of the cultures involved, our approach relies on the ability of people from within each culture to recognize an (etic) common feature in their own lived (emic) experience.…”

“…To overcome these limitations, we used the Response Process Evaluation (RPE) method [ 14 , 15 ], which provides a way to iteratively test if revisions improve the interpretability of an item both within and across cultures. As a form of web-probing, the RPE method uses meta-surveys consisting of several probes about survey items that prompt participants to explain two parts of their response process: their interpretation of the survey item and their rationale for selecting a response option.…”

Knowing What We're Talking About

“…The other five questions build upon this first question, firmly placing comprehensive construct definitions at the core of empirical research. Similarly, DCT specifications can be used to document the epistemic iteration that occurs when studying participants' response processes using Response Process Evaluation (Wolf et al, 2023).…”