Journals in Economic Sciences: Paying Lip Service to Reproducible Research?

Abstract

The findings of numerous replication studies in economics have raised serious concerns regarding the credibility and reliability of published applied economic research. Literature suggests that economic research often is not replicable because (i) only a small proportion of journals in the field have implemented functional policies on the disclosure of employed datasets and program code, (ii) authors frequently do not comply with these data policies and (iii) editorial offices do not ensure that these policies are enforced. In this paper, we focus on the aspect last mentioned. We empirically evaluate 599 articles published in 37 journals with a data availability policy. We present the share of articles that fall under a data policy, because replication data is needed to verify the published results. Afterwards, we check the journal data archives and supplemental information section of each article for the availability of replication files. For a reduced sub-sample of 245 data-based articles, we check in depth whether the replication files we found are compliant with the requirements of the journal’s respective data policy. Thereby, we are able to determine how much journals in economic sciences enforce their data policies. Our findings suggest a mixed picture: While some journals achieve high compliance rates, a significant share of journals only sporadically provides replication files for data-based research papers. Overall, 47.5% of all articles analysed honour the data policy of the respective journal. Our findings also provide evidence that voluntary data policies are not effective in fostering replicable research.

Beware ‘persuasive communication devices’ when writing and reading scientific articles

“…Superficial engagement with open research practices (such as underspecifying a pre-registered study) can also be used to mislead readers and reviewers about the strength of research (device 19; open research washing). For instance, one study of data availability in economics found that only 47.5% of articles in journals with a data-availability policy actually complied with the policy ( Vlaeminck and Podkrajac, 2017 ; see also Tedersoo et al, 2021 ).…”

Dawning of a new age? Economics journals’ data policies on the test bench

“…Of those 58 journals, 34 had a mandatory policy (58.6%). Chang and Li (2015) and Vlaeminck and Podkrajac (2017) have examined how voluntary and mandatory data policies perform. The authors concluded that mandatory data policies perform better, because the probability to find the data necessary to conduct reproductions is higher than for journals with voluntary data policies.…”

Supported Open Learning and Decoloniality: Critical Reflections on Three Case Studies

“…The literature on openwashing shows that at least some of the initiatives using the branding of "open" do not meet the criteria that most open education advocates would advance [22,23]. In the interest of retaining a critical perspective, it is fair to ask in whose interest "openness" operates [24]. At the least, we must be circumspect about grand claims of decolonisation through OER.…”