Recommendations for increasing replicability in psychology.

Abstract

Replicability of findings is at the heart of any empirical science. The aim of this article is to move the current replicability debate in psychology towards concrete recommendations for improvement. We focus on research practices but also offer guidelines for reviewers, editors, journal management, teachers, granting institutions, and university promotion committees, highlighting some of the emerging and existing practical solutions that can facilitate implementation of these recommendations. The challenges for improving replicability in psychological science are systemic. Improvement can occur only if changes are made at many levels of practice, evaluation, and reward.This target paper is the result of an Expert Meeting on 'Reducing non-replicable findings in personality research' in Trieste, Italy, July 14-16, 2012, financed by the European Association of Personality Psychology (EAPP) in the recognition of the current debate on insufficient replicability in psychology and medicine.

Playing with Data—Or How to Discourage Questionable Research Practices and Stimulate Researchers to Do Things Right

“…Pre-registration of the method of the research through peer review after which the journal guarantees publication irrespective of the outcome is expected to be effective to reduce QRPs of many sorts (e.g., Wagenmakers et al, 2012). Replication of research is often rightly advocated (Asendorpf et al, 2013a;Carpenter, 2012;Hartshorne & Schachner, 2012;Pashler & Wagenmakers, 2012;Yong, 2012), although one must realize that replication is most effective in controlled experiments done inside a lab. Exploration of one's data is a necessary research strategy in many research fields but may also invite accepting coincidental results that fail to be found in new data sets.…”

“…One topic is to teach the importance and the technicalities of data management, including data collection, administration, storage, and publication and to try make students realize that data sharing can be highly productive. The other topic is to stimulate in students a heightened awareness of the importance of understanding how to use methodology and statistics well in data analysis (Asendorpf et al, 2013a), and often the key is to learn to recognize when one needs expert help.…”

“…The scientific status of research results can only be maintained when others are granted the possibility to check whether results are correct and not based on QRPs. Journals should keep an eye on the match between the available data and the results presented in an article (see Lamont, 2009, for analysis of peer review, and Asendorpf et al, 2013a;Lilienfeld, 2012b;Mahoney, 1977;Nosek & Bar-Anan, 2012;Simmons et al, 2011, for recommendations on journal publication policy). This is why the availability of data is of the utmost importance: Without it, other researchers are destined to believe the researcher on his word without the possibility of verification.…”

“…Often inspired by psychological researchers, a quickly growing body of literature on errors psychological (and other) researchers make in their data collection, data analysis and data storage concentrates on the categorization of the errors, the consequences the errors have on the results of statistical analyses, the development of statistical methods for identifying the errors, and dos and don'ts that should spur researchers to do things right (e.g., Asendorpf et al, 2013a;Bakker, Van Dijk, & Wicherts, 2012;Fanelli, 2009;Ioannidis & Trakalinos, 2007;John, Loewenstein, & few publications focus on research policy needed to improve quality of research based on the analysis of empirical data (Anderson, 2014;Fanelli, 2013;Levelt et al, 2012). I argue that more than suggesting to researchers how they must do their research, policy measures at the institutional level are needed to reduce errors that researchers make due to incorrect methodological decisions and the incorrect use of statistical methods, and to stimulate the correct use of methodology and statistics.…”

“…Studies of QRPs are useful to determine different QRPs' prevalences (John et al, 2012) and may suggest consequences that inform policy and training practices (Asendorpf et al, 2013a), but also regularly raise the issue of intention. Establishing intention with individual researchers is relevant for assigning blame but may not be effective to reduce QRPs in general.…”