Statistical reporting errors in economics

Abstract

We developed a tool that scrapes and interprets statistical values (DORIS) to analyze reporting errors, which occur if the eye-catcher depicting the level of statistical significance is inconsistent with the reported statistical values. Using 578,132 tests from the top 50 economics journals, we find that 14.88 % of the articles have at least one strong error in the main tests. Our pre-registered analysis suggests that mandatory data and code availability policies reduce the prevalence of strong errors, while suggestive indication of a reversed effect is found for top 5 journals. Integrating DORIS into the review process can help improving article quality.