Improving transparency in observational social science research: A pre-analysis plan approach

Burlig, Fiona

doi:10.1016/j.econlet.2018.03.036

Cited by 53 publications

(9 citation statements)

References 38 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Whereas pre-registration has clear benefits for providing a verifiable documentation of the timing of study decisions in relation to data collection, the benefits of pre-registration becomes more debated when conducting secondary analyses of existing data sets. In particular, because some portion of the data has already been presented and because it can not be confirmed that the study team has not already explored the data set, pre-registration no longer serves to verify to readers that certain analyses were pre-planned and not data-driven (Burlig, 2018). Nonetheless, even with secondary analyses of existing data, registering an analyses plan prior to conducting an analysis can help keep a researcher focused and honest with themselves, even if it can not offer the same level of confidence to others.…”

Section: Discussionmentioning

confidence: 99%

Transparent reporting of hypotheses and analyses in behavioral medicine research: an audit of publications in 2018 and 2008

McVay

Cooper

Seoane

et al. 2021

Health Psychology and Behavioral Medicine

View full text Add to dashboard Cite

Objective: We aimed to document the use of transparent reporting of hypotheses and analyses in behavioral medicine journals in 2018 and 2008. Design: We examined a randomly selected portion of articles published in 2018 and 2008 by behavioral medicine journals with the highest impact factor, excluding manuscripts that were reviews or purely descriptive. Main Outcome Measures: We coded whether articles explicitly stated if the hypotheses/outcomes/analyses were primary or secondary; if study was registered/pre-registered; if 'exploratory' or a related term was used to describe analyses/aims; and if power analyses were reported. Results: We coded 162 manuscripts published in 2018 (87% observational and 12% experimental). Sixteen percent were explicit in describing hypotheses/outcomes/analyses as primary or secondary, 51% appeared to report secondary hypotheses/ outcomes/analyses but did not use term 'secondary,' and 33% were unclear. Registration occurred in 14% of studies, but 91% did not report which analyses were registered. 'Exploratory' or related term was used in 31% of studies. Power analyses were reported in 8% of studies. Compared to 2008 (n = 120), studies published in 2018 were more likely to be registered and less likely to be unclear if outcomes were primary or secondary. Conclusions: Behavioral medicine stakeholders should consider strategies to increase clarity of reporting, and particularly details that will inform readers if analyses were pre-planned or post-hoc. Study registration: https://osf.io/39ztn ARTICLE HISTORY

show abstract

Section: Discussionmentioning

confidence: 99%

Transparent reporting of hypotheses and analyses in behavioral medicine research: an audit of publications in 2018 and 2008

McVay

Cooper

Seoane

et al. 2021

Health Psychology and Behavioral Medicine

View full text Add to dashboard Cite

show abstract

“…A first, crucial set of norms speak to what, exactly, a complete PAP should contain and how PAPs should be adapted for observational studies, which comprise the majority of research projects undertaken in political science and economics (Burlig 2018;Jacobs 2020). Although several publications provide recommendations for what authors should (and need not) include in their PAPs (McKenzie 2012;Glennerster and Takavarasha 2013;Piñeiro and Rosenblatt 2016;Kern and Gleditsch 2017;Christensen, Freese, and Miguel 2019;Chen and Grady 2019;Duflo et al 2020), there are no universally agreed upon rules in either discipline for what a comprehensive PAP should look like.…”

Section: The Importance Of Complementary Norms and Institutionsmentioning

confidence: 99%

Pre-Analysis Plans: An Early Stocktaking

Ofosu¹,

Posner²

2021

Perspect. polit.

View full text Add to dashboard Cite

Pre-analysis plans (PAPs) have been championed as a solution to the problem of research credibility, but without any evidence that PAPs actually bolster the credibility of research. We analyze a representative sample of 195 PAPs registered on the Evidence in Governance and Politics (EGAP) and American Economic Association (AEA) registration platforms to assess whether PAPs registered in the early days of pre-registration (2011–2016) were sufficiently clear, precise, and comprehensive to achieve their objective of preventing “fishing” and reducing the scope for post-hoc adjustment of research hypotheses. We also analyze a subset of ninety-three PAPs from projects that resulted in publicly available papers to ascertain how faithfully they adhere to their pre-registered specifications and hypotheses. We find significant variation in the extent to which PAPs registered during this period accomplished the goals they were designed to achieve. We discuss these findings in light of both the costs and benefits of pre-registration, showing how our results speak to the various arguments that have been made in support of and against PAPs. We also highlight the norms and institutions that will need to be strengthened to augment the power of PAPs to improve research credibility and to create incentives for researchers to invest in both producing and policing them.

show abstract

“…Present in many fields (Fraser et al. 2018 ), the movement has also reached the social sciences and, more recently, political science (Burlig 2018 ; Freese and Peterson 2017 ; Monogan 2013 ; Wuttke 2019 ).…”

Section: Registered Reports: a Promising Approach To Overcoming Selec...mentioning

confidence: 99%

Special Issue Introduction

Bucher

Stroppe

Burger

et al. 2022

Polit Vierteljahresschr

View full text Add to dashboard Cite

The GLES Open Science Challenge 2021 was a pioneering initiative in quantitative political science. Aimed at increasing the adoption of replicable and transparent research practices, it led to this special issue. The project combined the rigor of registered reports—a new publication format in which studies are evaluated prior to data collection/access and analysis—with quantitative political science research in the context of the 2021 German federal election. This special issue, which features the registered reports that resulted from the project, shows that transparent research following open science principles benefits our discipline and substantially contributes to quantitative political science. In this introduction to the special issue, we first elaborate on why more transparent research practices are necessary to guarantee the cumulative progress of scientific knowledge. We then show how registered reports can contribute to increasing the transparency of scientific practices. Next, we discuss the application of open science practices in quantitative political science to date. And finally, we present the process and schedule of the GLES Open Science Challenge and give an overview of the contributions included in this special issue.

show abstract

Improving transparency in observational social science research: A pre-analysis plan approach

Cited by 53 publications

References 38 publications

Transparent reporting of hypotheses and analyses in behavioral medicine research: an audit of publications in 2018 and 2008

Transparent reporting of hypotheses and analyses in behavioral medicine research: an audit of publications in 2018 and 2008

Pre-Analysis Plans: An Early Stocktaking

Special Issue Introduction

Contact Info

Product

Resources

About