Open-ended versus Closed Probes: Assessing Different Formats of Web Probing

Abstract

The method of web probing integrates cognitive interviewing techniques into web surveys and is increasingly used to evaluate survey questions. In a usual web probing scenario, probes are administered immediately after the question to be tested (concurrent probing), typically as open-ended questions. A second possibility of administering probes is in a closed format, whereby the response categories for the closed probes are developed during previously conducted qualitative cognitive interviews. Using closed probes has several benefits, such as reduced costs and time efficiency, because this method does not require manual coding of open-ended responses. In this article, we investigate whether the insights gained into item functioning when implementing closed probes are comparable to the insights gained when asking open-ended probes and whether closed probes are equally suitable to capture the cognitive processes for which traditionally open-ended probes are intended. The findings reveal statistically significant differences with regard to the variety of themes, the patterns of interpretation, the number of themes per respondent, and nonresponse. No differences in number of themes across formats by sex and educational level were found.

Measurement invariance in the social sciences: Historical development, methodological challenges, state of the art, and future perspectives

“…The implementation in web surveys facilitates the recruitment of respondents, speeds up the data collection (Neuert et al, 2021), and easily increases sample sizes when compared to cognitive interviewing (Meitinger and Behr, 2016). This allows for an evaluation of the prevalence of themes and can explain the response patterns of specific subpopulations.…”

How to Design, Implement, and Analyse a Survey

“…The discussion of statistical models only includes basic information that is required to make decisions on question design (Chapter 3). It does not describe how to apply statistical methods, for which there are multiple sources on descriptive and multivariate statistics to analyse survey data, including econometrics textbooks (Cameron and Trivedi, 2005;Nardi, 2018) or the Sage series on quantitative applications in the social sciences, which provides in-depth explanations of close to 200 statistical methods.…”

“…Answering research questions can require descriptive or multivariate analyses (Nardi, 2018). Descriptive analyses provide frequencies (what percentage of respondents did x) or correlations (what activities are associated with other variables).…”

“…An experimental alternative to conducting cognitive testing in person or via a video link is web probing (Behr et al, 2017;Fowler and Willis, 2020;Neuert et al, 2021). A short online version of up to nine questions can be cognitively tested by following each question with a probe that asks for information such as why a specific response was selected out of several options, the respondent's understanding of a term, or for details on what the respondent was thinking when answering a question.…”

Implications of Online Self-Diagnosis in Psychiatry

“…The range and order of response alternatives may affect individual perception and the identification of risk 88 . The method used for probing online, open-ended versus closed questioning may lead to different results 89 . The inclusion of open-ended probes online may cause some people to stop, backtrack, or change prior answers 90 .…”