Acknowledging User Requirements for Accuracy in Computational Chemistry Benchmarks

Abstract

Computational chemistry has become an important complement to experimental measurements. In order to choose among the multitude of the existing approximations, it is common to use benchmark data sets, and to issue recommendations based on numbers such as mean absolute errors. We argue, using as an example band gaps calculated with density functional approximations, that a more careful study of the benchmark data is needed, stressing that the user's requirements play a role in the choice of an appropriate method. We also appeal to those who measure data capable of being used as a reference, to publish error estimates. We show how the latter can affect the judgment of approximations used in computational chemistry.

Should We Gain Confidence from the Similarity of Results between Methods?

“…For the band gap data set, we found that similarity of the density functional results had only a marginal impact on improving the prediction accuracy. This is consistent with previous findings that the differences between density functional approximations are less important when considering the error distributions [1,2], or taking into account experimental uncertainty [19].…”

DFT exchange: sharing perspectives on the workhorse of quantum chemistry and materials science

“…(3.3.9) Savin : True, the superiority of one DFA over another can be significantly diminished after taking into account the uncertainty in the reference data. 340 (3.3.10) Barone : Ideally a DFA should provide accurate results for a broad set of molecular properties. However, from a pragmatic point of view, there is a difference between specialized and broadly applicable DFAs and the choice between the two classes depends on the problem at hand.…”

“…True, the superiority of one DFA over another can be significantly diminished after taking into account the uncertainty in the reference data. 340 3.3.10 Barone. Ideally, a DFA should provide accurate results for a broad set of molecular properties.…”

“…True, the superiority of one DFA over another can be significantly diminished after taking into account the uncertainty in the reference data. 340 …”

The transferability limits of static benchmarks

“…Savin and Pernot have recently highlighted some shortcomings of benchmark studies. 5 Naturally, if the predefined set of molecules is small, it will only be representative for a small region of chemical space. However, for any size of the set, it will be important to know (1) whether even the region of reliable applicability is contiguous at all and (2) whether the boundaries of the region can be known for some predefined accuracy required for a meaningful result.…”