Sources of Variance in the Accuracy of Interviewer Observations

Abstract

In face-to-face surveys, interviewer observations are a cost-effective source of paradata for nonresponse adjustment of survey estimates and responsive survey designs. Unfortunately, recent studies have suggested that the accuracy of these observations can vary substantially among interviewers, even after controlling for household-, area-, and interviewer-level characteristics, limiting their utility. No study has identified sources of this unexplained variance in observation accuracy. Motivated by theoretical expectations from the observer bias literature, this study analyzed more than 45,000 open-ended justifications provided by interviewers in the U.S. National Survey of Family Growth (NSFG) for their observations on two key features of all sampled NSFG households: presence of children and expected probability of household response. The study finds that variability among interviewers in the cues used to record these observations (evident from the open-ended justifications) explains much of the previously unexplained variance in observation accuracy.

The imprisonment-extremism nexus: Continuity and change in activism and radicalism intentions in a longitudinal study of prisoner reentry

“…Follow-up interviews were facilitated using an extensive contact card generated from the in-prison interview and a client relationship management database to organize reminders, appointments, calls, and notes. The retention rates for these interviews were 66.3% ( N = 532) and 64.1% ( N = 514), consistent with large-scale studies of reentry [ 48 , 49 ]. Transient lifestyles, legal cynicism, and limited support systems make former prisoners a hard-to-reach population, and a complete description of post-release collection procedures is provided by Fahmy and colleagues [ 48 ].…”

“…The retention rates for these interviews were 66.3% ( N = 532) and 64.1% ( N = 514), consistent with large-scale studies of reentry [ 48 , 49 ]. Transient lifestyles, legal cynicism, and limited support systems make former prisoners a hard-to-reach population, and a complete description of post-release collection procedures is provided by Fahmy and colleagues [ 48 ].…”

“…Despite its significance, there are relatively few studies of activism and radicalization in prison, and many such studies depend on the views of correctional employees rather than prisoners themselves. In large part, this is due to the difficulty in gaining access to prisoners and tracking them upon release [ 48 , 49 ]. In a comprehensive review of the prevalence of Islam in U.S. prisons, Hamm notes that while there has been a dramatic increase in conversions to Islam among U.S. prisoners, it is hard to determine whether such conversions have led to increased or decreased terror activity, given the interventions to reduce such activity [ 50 ].…”

Characterizing low effort responding among young African adults recruited via Facebook advertising

“…Facebook’s advertising platform provides not only a recruitment tool but also a sampling frame—especially since no sampling frame of young African adults with access to the internet exists [ 67 ]. Although Facebook provides several options to reach a specified audience based on objectives such as brand awareness, reach, engagement, conversions, and traffic, I chose the traffic campaign objective based on evidence of effectiveness in prior studies [ 67 ]. According to Facebook, this campaign’s objective is to “send more people to a destination such as a website, app, or Messenger conversation.”…”

Understanding ‘Don't know’ answers to survey questions - an International comparative analysis using interview paradata

“…In the 2014 ESS there were 2,108 interviewers each completing an average of 19 interviews (ranging from an average of 8 interviews in the Czech Republic to an average of 24 in Switzerland). Whilst research by Sinibaldi et al, (2013) found only limited inconsistencies in observational data generated by differences between survey interviewers, other research has shown how the quality and consistency of responses during an interview can vary between observers (Clark et al, 2009;Loosveldt, 1999;West and Li 2019). A related issue is that the paradata about how the respondent answered and engaged with questions is produced by the interviewer only once the interview has been completed.…”