New method to compute R _complete enables maximum likelihood refinement for small datasets

Abstract

The crystallographic reliability index R complete is based on a method proposed more than two decades ago. Because its calculation is computationally expensive its use did not spread into the crystallographic community in favor of the cross-validation method known as R free . The importance of R free has grown beyond a pure validation tool. However, its application requires a sufficiently large dataset. In this work we assess the reliability of R complete and we compare it with k-fold cross-validation, bootstrapping, and jackknifing. As opposed to proper cross-validation as realized with R free , R complete relies on a method of reducing bias from the structural model. We compare two different methods reducing model bias and question the widely spread notion that random parameter shifts are required for this purpose. We show that R complete has as little statistical bias as R free with the benefit of a much smaller variance. Because the calculation of R complete is based on the entire dataset instead of a small subset, it allows the estimation of maximum likelihood parameters even for small datasets. R complete enables maximum likelihood-based refinement to be extended to virtually all areas of crystallographic structure determination including high-pressure studies, neutron diffraction studies, and datasets from free electron lasers. structure determination | reliability index | maximum likelihood refinement | overfitting | model bias T he quality of crystallographic models is described by several quality indicators. Both for small and macromolecular structure deposition, the crystallographic reliability index R1 must be provided (1, 2). It is calculated for the dataset H of observations and a structural model asDepending on the data-to-parameter ratio, R1 is affected by more or less severe overfitting (3, 4). To overcome this problem, cross-validation was introduced into crystallography (5-9). For cross-validation in crystallography, a certain fraction of the observations, typically 5-10%, are withheld as test set T and never used for model building and refinement. They are only used to calculate the reliability index R free :R free is much less affected by overfitting and since its introduction it has gained importance beyond validation of the structural model. It is used to optimize weights for restrained refinement (4, 10-13). The concept of R free paved the way for maximum likelihood methods in crystallography. It was shown that the estimation of maximum likelihood parameters based on the test set T provides much better accuracy than that based on the data used during refinement (14-16).Cross-validation reduces the bias of a statistic (17, 18) but can show large variance, especially when T is small (8, 17). The relative error of the crystallographic R free was established as σðR free Þ = R free = ffiffiffiffiffiffiffiffi ffi 2jTj p (19). The test set should hold at least 500 data points so that σðR free Þ=R free ≤ 0.032. Maximum likelihood methods estimate parameters in resolution bins, and a total of jTj ...

Structural Implications of Homopyrimidine Base Pairs in the Parallel‐Stranded d(YGA) Motif

“…Data collection and refinement statistics are given in Ta ble 1. Because of the very small number of reflections for the R2-53-C8C11c rystals, we also performed R complete analysis [30,31] using 250 independent test sets. This was carried out essentially as implemented in PDB-REDO, [32] but with phenix.refine used for the parallel refinements.…”

Model Selection for Biological Crystallography

“…R free is often incorporated into model selection methods [49,50]. But if R free is exploited both to select model type and to estimate model type errors [51], it may fall prey to precisely the problem it was designed to address: selecting an overly complex model type to fit spurious characteristics of ambient noise.…”

Protein structure determination by electron diffraction using a single three-dimensional nanocrystal

“…Like R work , it is likely to be affected by model bias. R complete was calculated afterwards according to standard procedures with a 0.2% testset size (Luebben & Gruene, 2015). Briefly, all nonmeasured observations were first removed from the reflection file with SFTOOLS (Winn et al, 2011).…”

“…‡ We present R1 and R complete instead of R work and R free . For less than 10 000 unique reflections R complete is preferred over R free , since it is calculated from all reflections (Brü nger, 1997; Luebben & Gruene, 2015). Since all structure factors are used, this in turn leads to a more robust calculation than R free .…”