Research participant selection in non-English language questionnaire pretesting: findings from Chinese and Korean cognitive interviews

Abstract

In survey research, cognitive interviewing has been widely used as a tool for pretesting to improve the quality of questionnaires. However, the cognitive interviewing method relies on interview participants' ability to articulate their thought process by answering the probe questions asked during the interview. Including participants who exhibit difficulty in understanding the probe question as intended by the researcher and difficulty in providing the appropriate answer pertinent to the probes affects the quality of cognitive interview outcome. Using main interview and its paradata from 258 non-English language (Chinese and Korean) cognitive interviews which pretest the American Community Survey questionnaire, this paper examines how levels of participant difficulty exhibited during the non-English cognitive interview affect the interview outcome. Furthermore, this paper provides practical guidelines about participant selection to balance interview quality and cost for pretesting translated survey questionnaires. We found that interviews with participants who exhibited greater difficulty were lengthier and did not add more value to the findings; in addition, those participants were less likely to complete any questionnaires in the future.

Health Research Participation, Opportunity, and Willingness Among Minority and Rural Communities of Arkansas

“…12 A lack of cultural awareness from researchers and language barriers can also limit research opportunities by creating communication gaps between researchers and potential research participants. 13,14 While rural and minority populations are often cited as being underrepresented in research, population-based studies on health research participation have not been conducted. Furthermore, we do not fully understand the factors associated with their willingness or unwillingness to participate in research.…”

Exploring the Effects of Pre-interview Practice in Asian Language Cognitive Interviews

“…Instead, this may indicate that the respondents' demographic differences across the two conditions were a key factor in performance. That is, since those in the enhanced condition were more educated and younger, they could have performed better related to those characteristics (Park et al 2016).…”

“…Often additional probing questions are asked to gauge the respondent's understanding of the meaning or intent of specific questionnaire items to illuminate respondents' understanding of the survey questions (Blair and Brick 2010;Hughes 2004). Therefore, respondents' articulation ability, that is, verbal ability to express one's thoughts and opinions clearly, can affect their performance in cognitive interviews (Park et al 2016). …”

“…In U.S. based research, non-English cognitive interviews are often reported to be more challenging than those conducted in English due to different communicative norms across cultures and lack of understanding or familiarity with the cognitive interview task (Goerman et al 2007;Pan et al 2010;Park et al 2016). In particular, respondents who speak a variety of languages have been found to provide responses to cognitive interview probes which are not extremely helpful in evaluating survey questions.…”

Do we see through their eyes? Testing a bilingual questionnaire in education research using cognitive interviews

“…A wide range of participants is recommended for CI as it allows examining differences in question interpretation (Beatty and Willis 2007; Collins and Gray 2014; Willis 2005). Furthermore, selecting interviewees with higher levels of education is advised as they find it easier to detect and articulate potential problems (Ackermann and Blair 2006;Collins 2014;Park, Sha, and Olmsted 2016). Considering that all our sample members were academics with higher degrees, their analytic skills were likely to facilitate problem identification.…”