Narratives of Response Error From Cognitive Interviews of Survey Questions About Normative Behavior

Abstract

That rates of normative behaviors produced by sample surveys are higher than actual behavior warrants is well evidenced in the research literature. Less well understood is the source of this error. Twentyfive cognitive interviews were conducted to probe responses to a set of common, conventional survey questions about one such normative behavior: religious service attendance. Answers to the survey questions and cognitive probes are compared both quantitatively and qualitatively. Half of the respondents amended their answer during cognitive probing, all amendments indicating a lower rate of attendance than originally reported, yielding a statistically significant reduction in reported attendance. Narrative responses shed light onto the source of bias, as respondents pragmatically interpreted the survey question to allow themselves to include other types of religious behavior, to report on a more religious past, and discount current constraints on their religious behavior, in order to report aspirational or normative religious identities.

Lies, Damned Lies, and Survey Self-Reports? Identity as a Cause of Measurement Bias

“…Understanding survey measurement as an opportunity for identity performance helps to explain the occurrence of overreporting even onself-administered surveys. Brenner (forthcoming) demonstrates that costs can situationally constrain identity performance. The respondent who values his or her identity as a physically active individual may fail to perform it given costs like time (i.e., an hour at the gym) and money (i.e., an expensive gym membership).…”

“…Comparisons of survey reports to criterion measures (administrative records and diary-like measures) found self-reported rates of exercise and church attendance that were double the actual frequency of these behaviors (Brenner and DeLamater 2014, forthcoming). Finding high rates of over-reporting of normative behavior even in self-administered modes calls into question the conventional wisdom that this phenomenon is simply the mirror image of the underreporting of counternormative behavior rooted in impression management.…”

ICT skills measurement in social surveys: Can we trust self-reports?

“…Unexpectedly, they also detected a correlation between social approval and the tendency to underreport physical activity, but the magnitude of this association was small. Several other studies have stressed the role of social desirability in the overreporting of physical activity, and have found different propensities to overreport for subgroups distinguished by sex, age, education, weight, and other characteristics (Beyler and Beyler 2017;Ferrari et al 2007;Hill et al 1997;Nusser et al 2012;Sallis and Saelens 2000;Troiano et al 2008); or by the data collection procedures used (Brenner andDeLamater 2016, 2014).…”

“…This bias has been related to social desirability scores, and appears to differ by sex, eating habits (amount of fat consumed), and, potentially, education (Hebert et al 1995(Hebert et al , 1997(Hebert et al , 2002. In addition, there is evidence that socially desirable answers are often given when reporting voting behaviour (Bernstein et al 2001;Holbrook and Krosnick 2010) and religious service attendance (Brenner 2017;Brenner and DeLamater 2016). In these cases, overreporting is usually observed.…”

Toward a social psychology of survey methodology: An application of the approach and directions for the future

“…Identity salience is defined as the propensity of enacting the identity or the propensity of defining a situation as one relevant for the performance of the identity (Stryker & Serpe, ). If the respondent strongly values an identity, such as his or her religious identity, but does not often perform it (typically, given the day‐to‐day constraints on his or her time given work and family responsibilities [Brenner, , ]), the survey question may provide an opportunity to perform that prominent identity (forthcoming; Brenner, ).…”