Does Chemical Engineering Research Have a Reproducibility Problem?

Abstract

Concerns have been raised in multiple scientific fields in recent years about the reproducibility of published results. Systematic efforts to examine this issue have been undertaken in biomedicine and psychology, but less is known about this important issue in the materials-oriented research that underpins much of modern chemical engineering. Here, we relate a dramatic historical episode from our own institution to illustrate the implications of performing reproducible research and describe two case studies based on literature analysis to provide concrete information on the reproducibility of modern materials-oriented research. The two case studies deal with the properties of metal-organic frameworks (MOFs), a class of materials that have generated tens of thousands of papers. We do not claim that research on MOFs is less (or more) reproducible than other subfields; rather, we argue that the characteristics of this subfield are common to many areas of materials-oriented research. We conclude with specific recommendations for action by individual researchers, journal editors, publishers, and research communities.

Exploring the limits of adsorption-based CO2 capture using MOFs with PVSA – from molecular design to process economics

“…we discuss the findings in the results and discussion section. Uncertainty also exists in isotherm measurements [39][40][41][42][43] , the root causes of this are multifaceted encompassing material synthesis, apparatus, and method. The utility of a sensitivity analysis on this is contentious, as isotherms repeated on the same sample on the same apparatus typically show little variation.…”

Does repeat synthesis in materials chemistry obey a power law?

“…For the materials shown in Fig. 1 it is challenging to discuss reproducibility because the small number of replicates that have been reported did not always report the same quantitative properties as the original synthesis report (27). Among the materials listed in SI Appendix, Table S7, however, there are many replicate measurements of MOF surface areas, a quantity that can be readily compared between experiments.…”

Benchmarking binding energy calculations for organic structure-directing agents in pure-silica zeolites

“…Finally, we highlight that reproducible calculations are extremely important in the field of porous materials and beyond. 86,87 Benchmarks enable the development of high-throughput computation infrastructures by providing clear guidelines for simulating materials with high robustness. We support further dissemination of these ideas.…”