Yes, no, maybe so: the importance of cognitive interviewing to enhance structured surveys on respectful maternity care in northern India

Abstract

Quantitative survey findings are important in measuring health-related phenomena, including on sensitive topics such as respectful maternity care (RMC). But how well do survey results truly capture respondent experiences and opinions? Quantitative tool development and piloting often involve translating questions from other settings and assessing the mechanics of implementation, which fails to deeply explore how respondents understand survey questions and response options. To address this gap, we conducted cognitive interviews on survey questions (n = 88) adapted from validated RMC instruments used in Ethiopia, Kenya and elsewhere in India. Cognitive interviews with rural women (n = 21) in Madhya Pradesh, India involved asking the respondent the survey question, recording her response, then interviewing her about what the question and response options meant to her. We analysed the interviews to revise the tool and identify question failures, which we grouped into six areas: issues with sequencing, length and sensitivity; problematic response options; inappropriate vocabulary; temporal and spatial confusion; accessing different cognitive domains; and failure to resonate with the respondent’s worldview and reality. Although women tended to provide initial answers to the survey questions, cognitive interviews revealed widespread mismatch between respondent interpretation and question intent. Likert scale response options were generally incomprehensible and questions involving hypothetical scenarios could be interpreted in unexpected ways. Many key terms and concepts from the international RMC literature did not translate well and showed low resonance with respondents, including consent and being involved in decisions about one’s care. This study highlights the threat to data quality and the validity of findings when translating quantitative surveys between languages and cultures and showcases the value of cognitive interviews in identifying question failures. While survey tool revision can address many of these issues, further critical discussion is needed on the use of standardized questions to assess the same domains across contexts.

Novel approaches to measuring knowledge among frontline health workers in India: Are phone surveys a reliable option?

“…We assumed ASHAs responded to knowledge questions with government recommendations in mind, rather than voicing personal opinions; however, further cognitive testing could have examined if this were truly the case. Because of resource limitations, cognitive testing-whilst done for other phone surveys developed as part of this project-was not feasible to undertake here [53]. Cognitive testing enables researchers to evaluate whether quantitative survey questions are generating the information that the researchers intend by exploring whether the questions are accessing the same cognitive domains among respondents as expected by the survey developers [54].…”

The devil is in the detail: reflections on the value and application of cognitive interviewing to strengthen quantitative surveys in global health

“…We used cognitive interviewing to examine survey questions for rural central India, adapted from validated instruments to measure respectful maternity care used in Ethiopia, Kenya and elsewhere in India. This process illuminated extensive cognitive mismatch between the intent of the original questions and how women interpreted them, which would have compromised the validity of the survey’s findings ( Scott et al , 2019 ). Two examples are provided here.…”

Measuring experiences of facility-based care for pregnant women and newborns: a scoping review

“…While this indicates the possibility for an expanded use of validated instruments, it is also important to note that when quantitative instruments are translated between languages and cultures, even validated tools may require additional work such as cognitive interviewing to ensure data quality, cultural appropriateness of measures and the validity of findings. 26 This review highlights a pressing need for developing, or using if it already exists, validated instruments for measuring various domains of experience of care. The importance of developing a coordinated approach to appraising and communicating available evidence on better measurement in global maternal and newborn health has been discussed elsewhere, 27 our review, documenting the widespread use of multiple, non-validated instruments, provides further evidence to support this call to action.…”