The devil is in the detail: reflections on the value and application of cognitive interviewing to strengthen quantitative surveys in global health

Abstract

Cognitive interviewing is a qualitative research method for improving the validity of quantitative surveys, which has been underused by academic researchers and monitoring and evaluation teams in global health. Draft survey questions are administered to participants drawn from the same population as the respondent group for the survey itself. The interviewer facilitates a detailed discussion with the participant to assess how the participant interpreted each question and how they formulated their response. Draft survey questions are revised and undergo additional rounds of cognitive interviewing until they achieve high comprehension and cognitive match between the research team’s intent and the target population’s interpretation. This methodology is particularly important in global health when surveys involve translation or are developed by researchers who differ from the population being surveyed in terms of socio-demographic characteristics, worldview, or other aspects of identity. Without cognitive interviewing, surveys risk measurement error by including questions that respondents find incomprehensible, that respondents are unable to accurately answer, or that respondents interpret in unintended ways. This methodological musing seeks to encourage a wider uptake of cognitive interviewing in global public health research, provide practical guidance on its application, and prompt discussion on its value and practice. To this end, we define cognitive interviewing, discuss how cognitive interviewing compares to other forms of survey tool development and validation, and present practical steps for its application. These steps cover defining the scope of cognitive interviews, selecting and training researchers to conduct cognitive interviews, sampling participants, collecting data, debriefing, analysing the emerging findings, and ultimately generating revised, validated survey questions. We close by presenting recommendations to ensure quality in cognitive interviewing.

The impact of a direct to beneficiary mobile communication program on reproductive and child health outcomes: a randomised controlled trial in India

“…From a measurement perspective, qualitative findings affirmed those from cognitive interviews done to support the design of survey tools 24 25. While the National Family Health Survey-4 findings from 2015 to 16 suggest that the reported practice of exclusive breast feeding was 58% in Madhya Pradesh, our findings suggest a much lower prevalence (intervention: 10.8%; control: 10.4%) when additional probing questions are incorporated in the survey tool.…”

Assessing the reliability of phone surveys to measure reproductive, maternal and child health knowledge among pregnant women in rural India: a feasibility study

“…Survey questions were derived from expert review and large national survey tools, including the demographic and health survey for India and the Multiple Indicator Cluster Survey. Cognitive interviews were used to refine questions on respectful maternity care, a process described elsewhere 19 20 36. Ideally, the same process would have been followed for refining knowledge questions.…”

Development and validation of the Self-Efficacy in Addressing Menstrual Needs Scale (SAMNS-26) in Bangladeshi schools: A measure of girls’ menstrual care confidence

“…We conducted 21 cognitive interviews in iterative rounds (2–6 interviews per round) with girls in Classes 5–10 in the same urban and rural schools where we had conducted FGDs. This iterative approach enabled us to identify and address issues with items and test revised versions until we achieved a good match between each item’s intent and participants’ interpretations [ 21 ]. We purposively selected girls from the master list of eligible students who had not previously participated in FGDs and according to time since menarche to ensure a range of familiarity with menstrual care.…”