Bias From Potentially Mischievous Responders on Large-Scale Estimates of Lesbian, Gay, Bisexual, or Questioning (LGBQ)–Heterosexual Youth Health Disparities

Abstract

Objectives. To determine how sensitive estimates of lesbian, gay, bisexual, or questioning (LGBQ)–heterosexual youth health disparities are to the presence of potentially mischievous responders. Methods. We used US data from the 2015 Youth Risk Behavior Survey, pooled across jurisdictions that included a question about sexual identity for a total sample of 148 960 students. We used boosted regressions (a machine-learning technique) to identify unusual patterns of responses to 7 screener items presumably unrelated to LGBQ identification, which generated an index of suspected mischievousness. We estimated LGBQ–heterosexual youth disparities on 20 health outcomes; then we removed 1% of suspected mischievous responders at a time and re-estimated disparities to assess the robustness of original estimates. Results. Accounting for suspected mischievousness reduced estimates of the average LGBQ–heterosexual youth health disparity by up to 46% for boys and 23% for girls; however, screening did not affect all outcomes equally. Drug- and alcohol-related disparities were most affected, particularly among boys, but bullying and suicidal ideation were unaffected. Conclusions. Including screener items in public health data sets and performing rigorous sensitivity analyses can support the validity of youth health estimates.

Large-Scale Estimates of LGBQ-Heterosexual Disparities in the Presence of Potentially Mischievous Responders: A Preregistered Replication and Comparison of Methods

“…Because mischievous responders are expected to provide low-frequency, extreme responses (Fan et al, 2006;Furlong, Fullchange, & Dowdy, 2017;Furlong, Sharkey, Bates, & Smith, 2004;Robinson-Cimpian, 2014), Cimpian and colleagues hypothesized that outcome items containing response options that were less frequently chosen (e.g., using heroin "40 or more times" in one's life) would be the items most affected by the removal of potentially mischievous responders. Cimpian et al (2018) found strong support for this hypothesis in the 2015 state and district sample, with large standardized Bs of 0.75 (ps < .001), among both males and females. Replicating this finding with the 2017 YRBS has implications for providing preregistered empirical support for this theory on how mischievous responders affect outcome estimates.…”

“…Thus, assumptions about which items and which specific item response-options are selected play a larger role in Methods 2 through 4. Regarding our prespecification of tempting item response-options for this analysis, the Cimpian et al (2018) analyses suggest a set of response-options to the screener items that are unusual and suggestive of likely mischievous responders. Based on that study, Table 4 presents the list of response-options we deem suggestive of mischievous responding in the 2017 YRBS.…”

“…As alluded to above, the current article has three components (separated as Studies). Study 1 is a direct replication of Cimpian et al (2018) examining how potentially mischievous responders may affect estimates of LGBQ-heterosexual disparities. Study 2 further replicates Cimpian et al (2018) by considering whether item-response extremity plays a role in those effects.…”

“Like Upgrading From a Typewriter to a Computer”: Registered Reports in Education Research

“…The preregistration process allowed the researchers to make timestamped predictions about those different approaches in advance of testing various combinations. Cimpian et al (2018) had previously examined the Youth Risk Behavior Survey and found that an unlikely percentage of boys reporting to be homosexual also reported a variety of unusual behaviors, for example, extreme heroin use. This study argued that that these unlikely behaviors were due to "mischievous responders," and this data integrity issue can lead to overstated risks to LGBTQ (lesbian, gay, bisexual, transgender, and queer or questioning) youth.…”

Did people really drink bleach to prevent COVID-19? A tale of problematic respondents and a guide for measuring rare events in survey data

“…Problematic respondents are not bound to specific modalities of survey sampling, such as specific national databases or particular online survey platforms, and are not limited to specific demographic populations. Rather, problematic respondent bias is an ubiquitous problem that requires mitigation in any type of survey (e.g Cimpian et al, 2018Fish and Russell, 2018; see Chandler et al, 2019), leading to an almost unanimous call by researchers who study survey data quality for including rigorous methodology to support the validity of estimates drawn from survey data (Jia, Konold, Cornell and Huang, 2018).…”