Misunderstanding standardized language in research interviews

Abstract

Leaving the interpretation of words up to participants in standardized survey interviews, aptitude tests, and experiment instructions can lead to unintended interpretation; more collaborative interviewing methods can promote uniform understanding. In two laboratory studies (a factorial experiment and a more naturalistic investigation), respondents interpreted ordinary survey concepts like 'household furniture' and 'living in a house' quite differently than intended in strictly standardized interviews, when the interpretation was left entirely up to them. Comprehension was more accurate when interviewers responded to requests for clarification with non-standardized paraphrased definitions, and most accurate when interviewers also provided clarification whenever they suspected respondents needed it.

Modeling Speech Disfluency to Predict Conceptual Misalignment in Speech Survey Interfaces

“…Not surprisingly, and consistent with prior findings (Schober & Conrad, 1997;Schober, Conrad & Fricker, 2004), participants fared very well overall at answering questions to straightforward mapping scenarios (mean accuracy of 94%), which varied little across conditions and age groups.…”

“…But if they are to be as effective as human interviewers, these machines would need to detect and predict cases of possible conceptual misalignment, where respondents need clarification of concepts in a question. Clarification has been shown to improve comprehension-and thus data quality-in survey interviews (Conrad & Schober, 2000;Schober & Conrad, 1997;Schober, Conrad & Fricker, 2004). Ideally, that clarification would be targeted and offered only when necessary, without being an intrusive burden, perhaps by modeling expectations about the speaker's needs.…”

Survey Interviews and New Communication Technologies

“…So, for example, if a respondent has a different interpretation of what it means to live in a household than the government agency asking the question, it might be useful for the interview to allow the kind of idiosyncratic interaction that would allow this discrepancy to be uncovered and for the respondent's report of how many people live in his house to conform with other respondents' interpretation. We have shown that data quality can be improved, although interviews take longer, on the telephone (Conrad and Schober, 2000;Schober and Conrad, 1997;Schober, Conrad, and Fricker, 2004), face to face (Schober et al, 2006), and in automated textual (Conrad, Schober, and Coiner, 2007) and speech (Ehlen, Schober, and Conrad, 2007;Schober et al, 2000) interviewing systems that allow nonstandardized clarification dialogue to take place.…”

The emergence of linguistic form in time

“…Once matchers realized this, they then shifted their gaze to the block on the next row up, which must have been the block to which the director was referring. Schober et al (2003) have tracked the gradual imposition of the listener's perspective over time in the brief conversations involved in telephone interviews and surveys. When we try to explain things to strangers, we often fail to grasp fully the difference between their perspectives and ours.…”