VarMatch: robust matching of small variant datasets using flexible scoring schemes

Sun, Chen; Medvedev, Paul

doi:10.1093/bioinformatics/btw797

Cited by 18 publications

(27 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A variety of approaches have been recently developed to address the challenges in variant representation. [9][10][11]21,22 Real Time Genomics (RTG) developed the comparison tool vcfeval, which introduced the idea of comparing variants at the level of the genomic haplotypes that the variants represent as a way to overcome the problems associated with comparing complex variants, where alternative yet equivalent variant representations can confound direct comparison methods. 9 Variant "normalization" tools help to represent variants in a standardized way (e.g., by left-shifting indels in repeats), but they demonstrated that "variant normalization" approaches alone were not able to reconcile different representations of many complex variants.…”

Section: Variant Representationmentioning

confidence: 99%

“…Similarly, VarMatch was developed to resolve alternate representations of complex variants, with additional ability to tune the matching parameters depending on the application. 10 Finally, hap.py includes a comparison tool to perform haplotype-based comparison of complex variants in addition to sophisticated functionality to stratify variant calls by type or region. 21 Note that representations include phasing information in these examples where it is necessary to unambiguously describe the variant.…”

Section: Variant Representationmentioning

confidence: 99%

“…First, benchmarking must consider that variants may be represented in multiple ways in the commonly used variant call format (VCF). [9][10][11][12] When comparing VCF files record by record, many of the putative differences are simply different representations of the same variant. Secondly, definitions for performance metrics such as true positive (TP), false positive (FP), and false negative (FN), which are key for the interpretation of the benchmarking results, are not yet standardized.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Best Practices for Benchmarking Germline Small Variant Calls in Human Genomes

Krusche¹,

Trigg

Boutros

et al. 2018

Preprint

122

189

View full text Add to dashboard Cite

Assessing accuracy of NGS variant calling is immensely facilitated by a robust benchmarking strategy and tools to carry it out in a standard way. Benchmarking variant calls requires careful attention to definitions of performance metrics, sophisticated comparison approaches, and stratification by variant type and genome context. The Global Alliance for Genomics and Health (GA4GH) Benchmarking Team has developed standardized performance metrics and tools for benchmarking germline small variant calls. This Team includes representatives from sequencing technology developers, government agencies, academic bioinformatics researchers, clinical laboratories, and commercial technology and bioinformatics developers for whom benchmarking variant calls is essential to their work. Benchmarking variant calls is a challenging problem for many reasons:• Evaluating variant calls requires complex matching algorithms and standardized counting, because the same variant may be represented differently in truth and query callsets.• Defining and interpreting resulting metrics such as precision (aka positive predictive value = TP/(TP+FP)) and recall (aka sensitivity = TP/(TP+FN)) requires standardization to draw robust conclusions about comparative performance for different variant calling methods.• Performance of NGS methods can vary depending on variant types and genome context; and as a result understanding performance requires meaningful stratification.• High-confidence variant calls and regions that can be used as "truth" to accurately identify false positives and negatives are difficult to define, and reliable calls for the most challenging regions and variants remain out of reach.We have made significant progress on standardizing comparison methods, metric definitions and reporting, as well as developing and using truth sets. Our methods are publicly available on GitHub (https://github.com/ga4gh/benchmarking-tools) and in a web-based app on precisionFDA, which allow users to compare their variant calls against truth sets and to obtain a standardized report on their variant calling performance. Our methods have been piloted in the precisionFDA variant calling challenges to identify the best-in-class variant calling methods within highconfidence regions. Finally, we recommend a set of best practices for using our tools and critically evaluating the results.

show abstract

Section: Variant Representationmentioning

confidence: 99%

Section: Variant Representationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Best Practices for Benchmarking Germline Small Variant Calls in Human Genomes

Krusche¹,

Trigg

Boutros

et al. 2018

Preprint

122

189

View full text Add to dashboard Cite

show abstract

“…Simplistic comparison methods that do not account for different representations of complex variants can cause methods that represent variants similarly to the benchmark to appear better than methods that represent variants in a different way. 20,21 The Global Alliance for Genomics and Health Benchmarking Team has developed sophisticated methods that account for these differences as long as the complex variant is fully called correctly in both the benchmark and test callsets. 8 Differences in representation can make up a significant fraction of putative FPs and FNs, so it is important to use these tools.…”

Section: Performance Metrics Can Vary Significantly When Comparing Tomentioning

confidence: 99%

Reproducible integration of multiple sequencing datasets to form high-confidence SNP, indel, and reference calls for five human genome reference materials

Zook

McDaniel

Parikh

et al. 2018

Preprint

View full text Add to dashboard Cite

Benchmark small variant calls from the Genome in a Bottle Consortium (GIAB) for the CEPH/HapMap genome NA12878 (HG001) have been used extensively for developing, optimizing, and demonstrating performance of sequencing and bioinformatics methods. Here, we improve and simplify the methods we use to integrate multiple sequencing datasets, with the intention of deploying a reproducible cloud-based pipeline for application to arbitrary human genomes. We use these reproducible methods to form high-confidence calls with respect to GRCh37 and GRCh38 for HG001 and 4 additional broadly-consented genomes from the Personal Genome Project that are available as NIST Reference Materials. Our new methods produce 17% more SNPs and 176% more indels than our previously published calls for HG001. We also phase 99.5% of the variants in HG001 and call about 90% of the reference genome with highconfidence, increased from 78% previously. Our calls only contain 108 differences from the Illumina Platinum Genomes calls in GRCh37, only 14 of which are ambiguous or likely to be errors in our calls. By comparing several callsets to our new calls, our previously published calls, and Illumina Platinum Genomes calls, we highlight challenges in interpreting performance metrics when benchmarking against imperfect high-confidence calls. Our new calls address some of these challenges, but performance metrics should always be interpreted carefully. Benchmarking tools from the Global Alliance for Genomics and Health are useful for stratifying performance metrics by variant type and genome context to elucidate strengths and weaknesses of a method. We also explore differences between comparing to high-confidence calls for the 5 GIAB genomes, and show that performance metrics for one pipeline are largely similar but not identical when comparing to the 5 genomes. Finally, to explore applicability of our methods for genomes that have fewer datasets, we form high-confidence calls using only Illumina and 10x Genomics, and find that they have more high-confidence calls but have a higher error rate. These newly characterized genomes have a broad, open consent with few restrictions availability of samples and data, enabling a uniquely diverse array of applications.

show abstract

“…The "strict matching" approach matches two indels if they share the same position, reference, and alternate alleles in two different entries in the VCF file. However, as demonstrated in [3], this approach fails to find equivalent indels that are not identical. The "distance based approach" treats two indels as equivalent if both have the same length 3 and occur within a certain distance such as ± 5 bp [5] or ± 25 bp [6].…”

Section: Introductionmentioning

confidence: 99%