Locating unregistered and unreported data for use in a social science systematic review and meta-analysis

Abstract

Meta-analysts rely on the availability of data from previously conducted studies. That is, they rely on primary study authors to register their outcome data, either in a study's text or on publicly available websites, and report the results of their work, either again in a study's text or on publicly accessible data repositories. If a primary study author does not register data collection and similarly does not report the data collection results, the meta-analyst is at risk of failing to include the collected data. The purpose of this study is to attempt to locate one type of metaanalytic data: findings from studies that neither registered nor reported the collected outcome data. To do so, we conducted a large-scale search for potential studies and emailed an author query request to more than 600 primary study authors to ask if they had collected eligible outcome data. We received responses from 75 authors (12.3%), three of whom sent eligible findings. The results of our search confirmed our proof of concept (i.e., that authors collect data but fail to register or report it publicly), and the meta-analytic results indicated that excluding the identified studies would change some of our substantive conclusions. Cost analyses indicated, however, a high price to finding the missing studies. We end by reaffirming our calls for greater adoption of primary study preregistration as well as data archiving in publicly available repositories.

The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews

“…Traditionally, the correlation between the interaction term and other variables is not reported and often must be requested directly from the original authors. Doing so is a high-risk endeavor given researchers' traditionally low response rate (Aguinis, Beaty, Boik, & Pierce, 2005;Polanin et al, 2020a), but the rise of Open Science and the concomitant Individual Participant Data (IPD) means that this information is increasingly available. Amalgamating IPD across multiple studies is usually referred to as a megaanalysis, and, as suggested here, can be used to supplement a standard meta-analysis (Boedhoe et al, 2019;Kaufmann, Reips, & Merki, 2016).…”

A systematic review and meta-analysis of interventions to decrease cyberbullying perpetration and victimization: An in-depth analysis within the Asia Pacific region

“…If the authors implemented a general violence or bullying prevention program but did not include a cyberbullying measure, we did not immediately exclude it. This procedure and the reasoning behind it have been explained by Polanin et al (37). They found that excluding the identified studies would change some substantive conclusions in their metaanalysis.…”

Pooling resources to enhance rigour in psychophysiological research: Insights from open science approaches to meta-analysis

“…One further limitation of this method is that it relies on cooperation from other researchers to locate, and sometimes re-analyse, unpublished data. These factors often mean that attempts to find unpublished studies returns a low yield (Polanin, Espelage, et al, 2020). Indeed, we (Saunders & Inzlicht, 2020) only uncovered 7 unpublished effect sizes-two of which were from our own laboratory.…”