Outcomes of the 2019 EMDataResource model challenge: validation of cryo-EM models at near-atomic resolution

Abstract

1This paper describes outcomes of the 2019 Cryo-EM Map-based Model Metrics Challenge 2 sponsored by EMDataResource (www.emdataresource.org). The goals of this challenge were (1) 3 to assess the quality of models that can be produced using current modeling software, (2) to 4 check the reproducibility of modeling results from different software developers and users, and 5(3) compare the performance of current metrics used for evaluation of models. The focus was on 6 near-atomic resolution maps with an innovative twist: three of four target maps formed a resolution 7 series (1.8 to 3.1 Å) from the same specimen and imaging experiment. Tools developed in 8 previous challenges were expanded for managing, visualizing and analyzing the 63 submitted 9 coordinate models, and several novel metrics were introduced. The results permit specific 10 recommendations to be made about validating near-atomic cryo-EM structures both in the context 11 of individual laboratory experiments and holdings of structure data archives such as the Protein 12 Data Bank. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM 13 models derived from these benchmark maps by 13 participating teams, representing both widely 14 used and novel modeling approaches. We also evaluate the pros and cons of the commonly used 15 metrics to assess model quality and recommend the adoption of multiple scoring parameters to 16 provide full and objective annotation and assessment of the model, reflective of the observed 17 density in the cryo-EM map. 18

Improving SARS-CoV-2 structures: Peer review by early coordinate release

“…MolProbity's version of traditional validations include outliers in bond lengths and angles, Ramachandran values, and side-chain rotamers. These are still extremely effective at resolutions better than $2.5 Å and do flag problems whenever they occur, but at lower resolutions, they are very often not seen because they have been tightly restrained to achieve stable, convergent refinement, usually without fixing the underlying problems (12).…”

“…So far, the most generally applicable MolProbity tool for 2.5-4 Å is Ca-BLAM (7,15), which uses Ca virtual angles to determine a robust backbone trace and then a virtual angle between successive backbone CO bond directions to find where peptide orientations are not compatible with the local Ca trace. CaBLAM flags incorrect peptide orientations even when Ramachandran outliers have been refined away, and in the recent cryo-EM model challenge, the CaBLAM score was found to have a higher correlation with match to target than any other criterion (12). In development is RNAprecis, a criterion to improve both modeling and validation of full-detail RNA conformations using features visible even at 3.5 Å .…”

Evolution of standardization and dissemination of cryo-EM structures and data jointly by the community, PDB, and EMDB

“…In 2019, we used our prior experience in setting up challenge events and aimed at modeling maps of two targets determined between 1.8 and 3.1 Å resolutions ( 64 ). This challenge was aimed at evaluating the quality of models produced by current modeling tools, the reproducibility of modeling results from different software developers and users, and the performance of current metrics used for evaluation of models.…”

League of Brazilian Bioinformatics: a competition framework to promote scientific training

“…Indeed, connecting students to problem solving challenges has been an approach that has demonstrated such an improvement in the ability to conduct research while expanding their opportunities to be absorbed by different sectors of industry and market (Pathanasethpong et al, 2019;Mason et al, 2009). With the gradual increase of professionals in Bioinformatics and the specialization that is requested, competitions for knowledge in this area began to arise (Kienzler and Fontanesi, 2017;Lawson et al, 2020;Connor et al, 2019;Pathanasethpong et al, 2019;Wang et al, 2018;Silver et al, 2016). Furthermore, hackathons have been applied in specific areas to explore public data, increase educational and innovation opportunities, create biological insights (2019 Model "Metrics" Challenge, Health Hackathons and NCBI's Virus Discovery Hackathon), and even solve new challenges and improve the state of the art (AlQuraishi, 2019;Zhou et al, 2019;Lawson et al, 2020).…”

“…With the gradual increase of professionals in Bioinformatics and the specialization that is requested, competitions for knowledge in this area began to arise (Kienzler and Fontanesi, 2017;Lawson et al, 2020;Connor et al, 2019;Pathanasethpong et al, 2019;Wang et al, 2018;Silver et al, 2016). Furthermore, hackathons have been applied in specific areas to explore public data, increase educational and innovation opportunities, create biological insights (2019 Model "Metrics" Challenge, Health Hackathons and NCBI's Virus Discovery Hackathon), and even solve new challenges and improve the state of the art (AlQuraishi, 2019;Zhou et al, 2019;Lawson et al, 2020). The experiment showed an effective development and reproducibility of the analyses performed during the competition by the teams.…”