Open exchange and epistemic progress.

Mahoney, Michael J.

doi:10.1037/0003-066x.40.1.29

Cited by 122 publications

(67 citation statements)

References 91 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This bias has been documented by investigations of the review process, which show that studies with findings that run counter to a priori hypotheses are less likely to receive favorable evaluations from reviewers and editors (Hubbard & Armstrong, 1997;Mahoney, 1985;Pfeffer, 2007;Sterling, Rosenbaum, & Weinkam, 1995). This tendency is compounded by the behavior of authors themselves, who are prone to withhold results that fail to support hypotheses (Rotton, Foos, Vanmeek, & Levitt, 1995), leading to what has been termed the ''file drawer problem'' (Rosenthal, 1979).…”

Section: Embrace Negative Resultsmentioning

confidence: 99%

The Presence of Something or the Absence of Nothing: Increasing Theoretical Precision in Management Research

Edwards

Berry

2010

Organizational Research Methods

209

210

View full text Add to dashboard Cite

In management research, theory testing confronts a paradox described by Meehl in which designing studies with greater methodological rigor puts theories at less risk of falsification. This paradox exists because most management theories make predictions that are merely directional, such as stating that two variables will be positively or negatively related. As methodological rigor increases, the probability that an estimated effect will differ from zero likewise increases, and the likelihood of finding support for a directional prediction boils down to a coin toss. This paradox can be resolved by developing theories with greater precision, such that their propositions predict something more meaningful than deviations from zero. This article evaluates the precision of theories in management research, offers guidelines for making theories more precise, and discusses ways to overcome barriers to the pursuit of theoretical precision. Keywords philosophy of science, quantitative research, theory developmentTheory testing in management research faces a paradox that confronts many social sciences. This paradox was articulated by Meehl (1967Meehl ( , 1978, who noted that in the hard sciences, such as physics and chemistry, improvements in research design lead to stronger tests of theories, subjecting them to increased risk of falsification. This phenomenon occurs because theories in the hard sciences can produce hypotheses that translate into predictions expressed as point values. As research designs become stronger (e.g., sample sizes are larger, measures have less error), estimates of point values have tighter confidence intervals, which increases the likelihood that hypotheses will be rejected. Hypotheses that survive these increasingly stringent conditions provide stronger corroboration of theories.In the soft sciences, such as management, stronger research designs yield weaker tests of theories. This paradox arises because theories in the soft sciences usually express predictions not as point values, but as directional statements, such as a positive or negative relationship between two variables. These predictions are tested by constructing a confidence interval not around some value predicted by the theory but instead around a null value indicating no effect. As research designs become stronger, confidence intervals around the null value become smaller, and the likelihood of concluding that the estimated effect falls on the predicted side of the null value approaches .50. To illustrate, suppose a theory predicts a positive relationship between two variables, and this prediction is tested with a sample of 380 cases and a p value of .05. Using a conventional two-tailed test, any correlation greater than .10 would be taken as support for the hypothesis. If a one-tailed test were used, taking into account the direction of the hypotheses, only 270 cases would be required to declare a correlation greater than .10 significant at p < .05. Hypotheses that accommodate such a broad range of values confront low hurdle...

show abstract

Section: Embrace Negative Resultsmentioning

confidence: 99%

The Presence of Something or the Absence of Nothing: Increasing Theoretical Precision in Management Research

Edwards

Berry

2010

Organizational Research Methods

209

210

View full text Add to dashboard Cite

show abstract

“…Related to this, direct replication of another's study procedures to confirm the results is uncommon in the social sciences (Collins, 1985;Mahoney, 1985;Schmidt, 2009). Neuliep and Crandall (1990;see also Madden, Easley, & Dunn, 1995;Neuliep & Crandall, 1993) found that 94% of journal editors agreed that "replication studies were not included as examples of research encouraged for submission in the editorial policy" (p. 87), and a large majority preferred to publish new findings rather than replications because the latter were deemed "not newsworthy" and a "waste of space.…”

Section: Novelty and Positive Results Are Vital For Publishability Bumentioning

confidence: 99%

Scientific Utopia

Nosek

Spies

Motyl

2012

Perspect Psychol Sci

1,061

421

View full text Add to dashboard Cite

An academic scientist's professional success depends on publishing. Publishing norms emphasize novel, positive results. As such, disciplinary incentives encourage design, analysis, and reporting decisions that elicit positive results and ignore negative results. Prior reports demonstrate how these incentives inflate the rate of false effects in published science. When incentives favor novelty over replication, false results persist in the literature unchallenged, reducing efficiency in knowledge accumulation. Previous suggestions to address this problem are unlikely to be effective. For example, a journal of negative results publishes otherwise unpublishable reports. This enshrines the low status of the journal and its content. The persistence of false findings can be meliorated with strategies that make the fundamental but abstract accuracy motive-getting it right-competitive with the more tangible and concrete incentive-getting it published. This article develops strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and biases.

show abstract

“…Attempts to replicate original research findings seem uncommon in psychology (Asendorpf et al, 2013;Mahoney, 1985;Makel, Plucker, & Hegarty, 2012; but also see Neuliep & Crandall, 1993a;Neuliep & Crandall, 1993b), but when replications are conducted, they produce weaker evidence in the large majority of cases, or even no evidence for the original findings in many cases (Asendorpf et al, 2013;Chang & Li, 2015;Ioannidis, 2005aIoannidis, , 2007Marsman et al, 2017;Mobley, Linder, Braeuer, Ellis, & Zwelling, 2013;Open Science Collaboration, 2015). In recent years, many psychologists have expressed their concerns about the size and the gravity of these problems, even referring to it as a replication crisis (Nosek & Lakens, 2015;Open Science Collaboration, 2015;Pashler & Harris, 2012;Spellman, 2015).…”

Section: Chapter 6 Abstractmentioning

confidence: 99%

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Veldkamp¹

2017

Preprint

View full text Add to dashboard Cite

THE HUMAN FALLIBILITY OF SCIENTISTSDealing with error and bias in academic researchRecent studies have highlighted that not all published findings in the scientific literature are trustworthy, suggesting that currently implemented control mechanisms such as high standards for the reporting of research methods and results, peer review, and replication, are not sufficient. In psychology in particular, solutions are sought to deal with poor reproducibility and replicability of research results. In this dissertation project I considered these problems from the perspective that the scien¬tific enterprise must better recognize the human fallibility of scientists, and I examined potential solutions aimed at dealing with human error and bias in psychological science. First, I studied whether the human fallibility of scientists is actually recognized (Chapter 2). I examined the degree to which scientists and lay people believe in the storybook image of the scientist: the image that scientists are more objective, rational, open-minded, intelligent, honest and communal than other human beings. The results suggested that belief in this storybook image is strong, particularly among scientists themselves. In addition, I found indications that scientists believe that scientists like themselves fit the storybook image better than other scientists. I consider scientist’s lack of acknowledgement of their own fallibility problematic, because I believe that critical self-reflection is the first line of defense against potential human error aggravated by confirmation bias, hindsight bias, motivated reasoning, and other human cognitive biases that could affect any professional in their work. Then I zoomed in on psychological science and focused on human error in the use of null the most widely used statistical framework in psychology: hypothesis significance testing (NHST). In Chapters 3 and 4, I examined the prevalence of errors in the reporting of statistical results in published articles, and evaluated a potential best practice to reduce such errors: the so called ‘co-pilot model of statistical analysis’. This model entails a simple code of conduct prescribing that statistical analyses are always conducted independently by at least two persons (typically co-authors). Using statcheck, a software package that is able to quickly retrieve and check statistical results in large sets of published articles, I replicated the alarmingly high error rates found in earlier studies. Although I did not find support for the effectiveness of the co-pilot model in reducing these errors, I proposed several ways to deal with human error in (psychological) research and suggested how the effectiveness of the proposed practices might be studied in future research. Finally, I turned to the risk of bias in psychological science. Psychological data can often be analyzed in many different ways. The often arbitrary choices that researchers face in analyzing their data are called researcher degrees of freedom. Researchers might be tempted to use these researcher degrees of freedom in an opportunistic manner in their pursuit of statistical significance (often called p-hacking). This is problematic because it renders research results unreliable. In Chapter 5 I presented a list of researcher degrees of freedom in psychological studies, focusing on the use of NHST. This list can be used to assess the potential for bias in psychological studies, it can be used in research methods education, and it can be used to examine the effectiveness of a potential solution to restrict oppor¬tunistic use of RDFs: study pre-registration. Pre-registration requires researchers to stipulate in advance the research hypothesis, data collection plan, data analyses, and what will be reported in the paper. Different forms of pre-registration are currently emerging in psychology, mainly varying in terms of the level of detail with respect to the research plan they require researchers to provide. In Chapter 6, I assessed the extent to which current pre-registrations restricted opportunistic use of the researcher degrees of freedom on the list presented in Chapter 5. We found that most pre-registrations were not sufficiently restrictive, but that those that were written following better guidelines and requirements restricted opportunistic use of researcher degrees of freedom considerably better than basic pre-registrations that were written following a limited set of guidelines and requirements. We concluded that better instructions, specific questions, and stricter requirements are necessary in order for pre-registrations to do what they are supposed to do: to protect researchers from their own biases.

show abstract

Open exchange and epistemic progress.

Cited by 122 publications

References 91 publications

The Presence of Something or the Absence of Nothing: Increasing Theoretical Precision in Management Research

The Presence of Something or the Absence of Nothing: Increasing Theoretical Precision in Management Research

Scientific Utopia

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Contact Info

Product

Resources

About