Life after P-Hacking

Simmons, Joseph P.; Nelson, Leif D.; Simonsohn, Uri

doi:10.2139/ssrn.2205186

Cited by 252 publications

(258 citation statements)

References 0 publications

Supporting

Mentioning

254

Contrasting

Unclassified

Order By: Relevance

“…By means of conceptual replication, the results therefore validate the conceptual priming finding by showing that it extends to social dilemmas involving structural inequality. Naturally, this conceptual replication does not render exact replications useless [43][44][45][46][47]. For instance, future research could investigate whether expertise effects ( [15], Study 9) also moderate the effect of intuition on cooperation under structural inequality.…”

Section: Resultsmentioning

confidence: 99%

“…Following recent sample size suggestions [47], a total of 250 HITs were created on Amazon Mechanical Turk ( [48], for validity of AMT as a research tool), with a stopping rule automatically enforced upon completion of these HITs. The HIT was named "Decision-making task" and the description of the task was as follows: "Work on a short task related to decision making (max.…”

Section: Methodsmentioning

confidence: 99%

“…As many scholars have noted [43][44][45][46][47], direct replications serve to investigate the reproducibility of a certain effect, whereas conceptual replication serves as a validation of a previously established effect. The research question here was whether or not intuition promotes cooperation in asymmetric contexts (i.e., as this has not been investigated before and is interesting due to the ability to disentangle cooperativeness from equality), thus a direct replication of Rand et al's Study 8 [15] would be irrelevant to the present context that involves asymmetry, despite being an interesting (and relevant) research question in itself.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Spontaneous Giving Under Structural Inequality: Intuition Promotes Cooperation in Asymmetric Social Dilemmas

Lotz

2014

SSRN Journal

View full text Add to dashboard Cite

The present research investigates the role of intuitive mental processing on cooperation in experimental games involving structural inequality. Results from an experiment using conceptual priming to induce intuitive mental processing provide the first evidence that cooperation is promoted by intuition in an asymmetric context that distributes the gains from cooperation unequally among a group. Therefore, the results extend our understanding of the cognitive underpinnings of human cooperation by demonstrating the robustness of intuitive cooperation in games involving structural inequality regarding asymmetric gains from cooperation. Additionally, the results provide the first successful conceptual replication of the intuition-cooperation link using conceptual priming, therefore also contributing to the debate about the validity of previous research in other contexts. Taken together, the present research contributes to the literature on psychological and institutional mechanisms that promote cooperation.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Spontaneous Giving Under Structural Inequality: Intuition Promotes Cooperation in Asymmetric Social Dilemmas

Lotz

2014

SSRN Journal

View full text Add to dashboard Cite

show abstract

“…A key issue with these choices is that researchers might use these so-called researcher degrees of freedom strategically in order to obtain statistically significant results Simmons et al, 2011). Opportunistic use of researcher degrees is commonly known as 'p-hacking' (Gelman & Loken, 2013;John et al, 2012;Simmons, Nelson, & Simonsohn, 2013;Simonsohn, Nelson, & Simmons, 2014a) and is problematic for two main reasons. First, p-hacking greatly increases the chances of finding a false positive result (DeCoster, Sparks, Sparks, Sparks, & Sparks, 2015;Ioannidis, 2005b;Simmons et al, 2011).…”

Section: Chaptermentioning

confidence: 99%

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Veldkamp¹

2017

Preprint

View full text Add to dashboard Cite

THE HUMAN FALLIBILITY OF SCIENTISTSDealing with error and bias in academic researchRecent studies have highlighted that not all published findings in the scientific literature are trustworthy, suggesting that currently implemented control mechanisms such as high standards for the reporting of research methods and results, peer review, and replication, are not sufficient. In psychology in particular, solutions are sought to deal with poor reproducibility and replicability of research results. In this dissertation project I considered these problems from the perspective that the scien¬tific enterprise must better recognize the human fallibility of scientists, and I examined potential solutions aimed at dealing with human error and bias in psychological science. First, I studied whether the human fallibility of scientists is actually recognized (Chapter 2). I examined the degree to which scientists and lay people believe in the storybook image of the scientist: the image that scientists are more objective, rational, open-minded, intelligent, honest and communal than other human beings. The results suggested that belief in this storybook image is strong, particularly among scientists themselves. In addition, I found indications that scientists believe that scientists like themselves fit the storybook image better than other scientists. I consider scientist’s lack of acknowledgement of their own fallibility problematic, because I believe that critical self-reflection is the first line of defense against potential human error aggravated by confirmation bias, hindsight bias, motivated reasoning, and other human cognitive biases that could affect any professional in their work. Then I zoomed in on psychological science and focused on human error in the use of null the most widely used statistical framework in psychology: hypothesis significance testing (NHST). In Chapters 3 and 4, I examined the prevalence of errors in the reporting of statistical results in published articles, and evaluated a potential best practice to reduce such errors: the so called ‘co-pilot model of statistical analysis’. This model entails a simple code of conduct prescribing that statistical analyses are always conducted independently by at least two persons (typically co-authors). Using statcheck, a software package that is able to quickly retrieve and check statistical results in large sets of published articles, I replicated the alarmingly high error rates found in earlier studies. Although I did not find support for the effectiveness of the co-pilot model in reducing these errors, I proposed several ways to deal with human error in (psychological) research and suggested how the effectiveness of the proposed practices might be studied in future research. Finally, I turned to the risk of bias in psychological science. Psychological data can often be analyzed in many different ways. The often arbitrary choices that researchers face in analyzing their data are called researcher degrees of freedom. Researchers might be tempted to use these researcher degrees of freedom in an opportunistic manner in their pursuit of statistical significance (often called p-hacking). This is problematic because it renders research results unreliable. In Chapter 5 I presented a list of researcher degrees of freedom in psychological studies, focusing on the use of NHST. This list can be used to assess the potential for bias in psychological studies, it can be used in research methods education, and it can be used to examine the effectiveness of a potential solution to restrict oppor¬tunistic use of RDFs: study pre-registration. Pre-registration requires researchers to stipulate in advance the research hypothesis, data collection plan, data analyses, and what will be reported in the paper. Different forms of pre-registration are currently emerging in psychology, mainly varying in terms of the level of detail with respect to the research plan they require researchers to provide. In Chapter 6, I assessed the extent to which current pre-registrations restricted opportunistic use of the researcher degrees of freedom on the list presented in Chapter 5. We found that most pre-registrations were not sufficiently restrictive, but that those that were written following better guidelines and requirements restricted opportunistic use of researcher degrees of freedom considerably better than basic pre-registrations that were written following a limited set of guidelines and requirements. We concluded that better instructions, specific questions, and stricter requirements are necessary in order for pre-registrations to do what they are supposed to do: to protect researchers from their own biases.

show abstract

“…Researchers have started to realize that especially in between-subject designs, a much larger number of participants have to be collected than psychologists were accustomed to if they desire to perform high-powered studies. Some researchers have suggested a minimum of 20 participants per condition (e.g., Simmons et al, 2011, later increased to a minimum of 50 participants per condition, Simmons, Nelson, & Simonsohn, 2013), but such well-intended suggestions are bad advice. Sample sizes should be determined on the basis of the expected size of the effect, the desired power, and the planned alpha level of the statistical test or on the basis of the desired width of the confidence interval (CI) surrounding the effect size estimate.…”

Section: Introductionmentioning

confidence: 99%

Performing High-Powered Studies Efficiently With Sequential Analyses

Lakens¹

2017

Preprint

View full text Add to dashboard Cite

Running studies with high statistical power, while effect size estimates in psychology are often inaccurate, leads to a practical challenge when designing an experiment. This challenge can be addressed by performing sequential analyses while the data collection is still in progress. At an interim analysis, data collection can be stopped whenever the results are convincing enough to conclude that an effect is present, more data can be collected, or the study can be terminated whenever it is extremely unlikely that the predicted effect will be observed if data collection would be continued. Such interim analyses can be performed while controlling the Type 1 error rate. Sequential analyses can greatly improve the efficiency with which data are collected. Additional flexibility is provided by adaptive designs where sample sizes are increased on the basis of the observed effect size. The need for pre-registration, ways to prevent experimenter bias, and a comparison between Bayesian approaches and null-hypothesis significance testing (NHST) are discussed. Sequential analyses, which are widely used in large-scale medical trials, provide an efficient way to perform high-powered informative experiments. I hope this introduction will provide a practical primer that allows researchers to incorporate sequential analyses in their research. Copyright © 2014 John Wiley & Sons, Ltd.Repeatedly analyzing results while data collection is in progress has many advantages. Researchers can stop the data collection when observed differences reach a desired confidence level or when unexpected data patterns occur that warrant a reconsideration of the aims of the study. When, after an interim analysis, the effect is smaller than expected, researchers might decide to collect more data or even stop collecting data for specific conditions. One could easily argue that psychological researchers have an ethical obligation to repeatedly analyze accumulating data, given that continuing data collection whenever the desired level of confidence is reached, or whenever it is sufficiently clear that the expected effects are not present, is a waste of the time of participants and the money provided by taxpayers. In addition to this ethical argument, designing studies that make use of sequential analyses are more efficient compared with not performing sequential analyses. Incorporating sequential analyses into the study design can easily reduce the sample size of studies by 30% or more.In psychology, sequential analyses are rarely, if ever, used. In recent years, researchers have been reminded of the fact that repeatedly analyzing data, and continuing the data collection when results are not significant, increases the likelihood of a Type 1 error, or a significant test result in the absence of any differences in the population (e.g., Simmons, Nelson, & Simonsohn, 2011). Flexibility in data collection does not lead to an inevitable increase in Type 1 errors. It is possible to repeatedly analyze data, and decide whether to continue or end the data...

show abstract

Life after P-Hacking

Cited by 252 publications

References 0 publications

Spontaneous Giving Under Structural Inequality: Intuition Promotes Cooperation in Asymmetric Social Dilemmas

Spontaneous Giving Under Structural Inequality: Intuition Promotes Cooperation in Asymmetric Social Dilemmas

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Performing High-Powered Studies Efficiently With Sequential Analyses

Contact Info

Product

Resources

About