How Big of a Problem is Analytic Error in Secondary Analyses of Survey Data?

Abstract

Secondary analyses of survey data collected from large probability samples of persons or establishments further scientific progress in many fields. The complex design features of these samples improve data collection efficiency, but also require analysts to account for these features when conducting analysis. Unfortunately, many secondary analysts from fields outside of statistics, biostatistics, and survey methodology do not have adequate training in this area, and as a result may apply incorrect statistical methods when analyzing these survey data sets. This in turn could lead to the publication of incorrect inferences based on the survey data that effectively negate the resources dedicated to these surveys. In this article, we build on the results of a preliminary meta-analysis of 100 peer-reviewed journal articles presenting analyses of data from a variety of national health surveys, which suggested that analytic errors may be extremely prevalent in these types of investigations. We first perform a meta-analysis of a stratified random sample of 145 additional research products analyzing survey data from the Scientists and Engineers Statistical Data System (SESTAT), which describes features of the U.S. Science and Engineering workforce, and examine trends in the prevalence of analytic error across the decades used to stratify the sample. We once again find that analytic errors appear to be quite prevalent in these studies. Next, we present several example analyses of real SESTAT data, and demonstrate that a failure to perform these analyses correctly can result in substantially biased estimates with standard errors that do not adequately reflect complex sample design features. Collectively, the results of this investigation suggest that reviewers of this type of research need to pay much closer attention to the analytic methods employed by researchers attempting to publish or present secondary analyses of survey data.

Analytic Errors in Analysis of Public Health Survey Data Are Avoidable

“…Consistent with my own experience as a peer reviewer, empirical studies have found this type of error to be common (1,2). A meta-analysis of 100 peer-reviewed journal articles that performed secondary analysis of data from nationally representative surveys with complex sample designs found 1,100 instances in which analytic errors may have occurred, and in 616 instances these errors were likely present (1).…”

“…Yet, as a peer reviewer of scientific journals, I regularly find that many studies that analyze survey data used inappropriate methods of estimation, known as analytic error (1,2). Some examples of these errors include not applying data weights, overlooking complex survey design, and not properly subsetting data when analyzing subpopulations.…”

Repeat participation in annual cross-sectional surveys of drug users and its implications for analysis

“…Primary health research involving complex sampling often employs inappropriate statistical approaches to inference, and often gives insufficient detail to provide methodological clarity [1, 2]. Related to this is the issue in repeated cross-sectional studies whereby pooled cross-sectional estimation in the presence of repeat responses from the same individuals can yield biased estimates and incorrect estimates of standard error if inappropriate statistical methodology is applied [2, 3].…”

“…Related to this is the issue in repeated cross-sectional studies whereby pooled cross-sectional estimation in the presence of repeat responses from the same individuals can yield biased estimates and incorrect estimates of standard error if inappropriate statistical methodology is applied [2, 3]. Typically, failure to account for such lack of independence in observations or clustering will underestimate standard errors, resulting in biased inference which in turn may lead to type I error [4, 5].…”

Odyssey in the evolution of a paleopathologist

“…Whether written word, painting or sculpture, those art forms provide a window to ancient perspectives of health and disease, if not actually necessarily reflecting the character of a given disease at the time (Mitchell, 2011). Access to historical information is complicated by variable accessibility of original documents, knowledge of word meanings at the time, conventions and imagery of those times and of course ability to accurately translate those documents (West et al, 2016). The artistic convention of flexed metacarpal phalangeal joints, for example, gives the false impression of disease-related ulnar deviation, suggesting presence of inflammatory arthritis, when it is simply the artistically adapted pose.…”