Best Practices for Benchmarking Germline Small Variant Calls in Human Genomes

Abstract

Assessing accuracy of NGS variant calling is immensely facilitated by a robust benchmarking strategy and tools to carry it out in a standard way. Benchmarking variant calls requires careful attention to definitions of performance metrics, sophisticated comparison approaches, and stratification by variant type and genome context. The Global Alliance for Genomics and Health (GA4GH) Benchmarking Team has developed standardized performance metrics and tools for benchmarking germline small variant calls. This Team includes representatives from sequencing technology developers, government agencies, academic bioinformatics researchers, clinical laboratories, and commercial technology and bioinformatics developers for whom benchmarking variant calls is essential to their work. Benchmarking variant calls is a challenging problem for many reasons:• Evaluating variant calls requires complex matching algorithms and standardized counting, because the same variant may be represented differently in truth and query callsets.• Defining and interpreting resulting metrics such as precision (aka positive predictive value = TP/(TP+FP)) and recall (aka sensitivity = TP/(TP+FN)) requires standardization to draw robust conclusions about comparative performance for different variant calling methods.• Performance of NGS methods can vary depending on variant types and genome context; and as a result understanding performance requires meaningful stratification.• High-confidence variant calls and regions that can be used as "truth" to accurately identify false positives and negatives are difficult to define, and reliable calls for the most challenging regions and variants remain out of reach.We have made significant progress on standardizing comparison methods, metric definitions and reporting, as well as developing and using truth sets. Our methods are publicly available on GitHub (https://github.com/ga4gh/benchmarking-tools) and in a web-based app on precisionFDA, which allow users to compare their variant calls against truth sets and to obtain a standardized report on their variant calling performance. Our methods have been piloted in the precisionFDA variant calling challenges to identify the best-in-class variant calling methods within highconfidence regions. Finally, we recommend a set of best practices for using our tools and critically evaluating the results.

Merqury: reference-free quality, completeness, and phasing assessment for genome assemblies

“…Moreover, k-mers naturally capture heterozygous insertion and deletion variants and are thus immune to the problems of calling these types of variants with a reference mapping approach. For example, consortiums such as the GA4GH exclude all variant calls within complex, repetitive regions of the genome 25 . In contrast, hap-mers inherently capture genetic context, regardless of the structural complexity surrounding them in the genome.…”

“…Alternative methods report phasing statistics from small variants (mostly SNPs) called with short-read mapping 8,[19][20][21][22] , or use benchmark genomes that have curated, phased variation call sets [23][24][25][26] . Both methods rely on a reference sequence as the primary source to detect heterozygous variations.…”

“…However, these reference sequences are incomplete and recent studies have demonstrated the shortcomings of the current human reference genome and variant call sets 8,20,27 . For example, the highly variable major histocompatibility complex (MHC) is excluded from the Genome in a Bottle (GIAB) or Global Alliance for Genomic Health (GA4GH) reference panel 25 , due to its repetitive nature and need of a specialized mapping strategy to account for the high allelic diversity 28 . Moreover, reference-guided strategies require significant manual curation and effort and will not scale as large cohort sequencing projects become common 29,30 .…”

A Diploid Assembly-based Benchmark for Variants in the Major Histocompatibility Complex

“…We evaluated the utility and accuracy of our benchmark MHC small variant set by comparing a DeepVariant v0.8 callset 18 from ~30x PacBio Sequel II 11 kb CCS reads to the benchmark, followed by manually curating putative false positives (FPs) and false negatives (FNs). When using hap.py with the vcfeval option to account for differences in representation of the many complex variants in the MHC 19 , there were 20074 TPs (matching 19320 calls in the benchmark VCF), 366 FPs, and 2176 FNs (of which 290 FPs and 260 FNs were genotyping errors or partial allele matches). To show our benchmark reliably identifies FPs and FNs, we manually curated 10 random genotyping errors or partial allele matches, as well as 10 random FPs and 10 random FNs that were not genotyping errors or partial allele matches.…”

Machine learning-based detection of insertions and deletions in the human genome

“…We subdivide each list into separate registers of insertion, deletion start, and deletion end breakpoints. Corresponding truth-query pairs are input to an app developed by the Global Alliance for Genomics and Health (GA4GH Benchmarking) made available on precisionFDA 35 . In a distance-based comparison, it categorizes calls as true positives, false positives, or false negatives, and uses this information to derive precision and recall scores.…”