No one tool to rule them all: prokaryotic gene prediction tool annotations are highly dependent on the organism of study

Abstract

Motivation The biases in CoDing Sequence (CDS) prediction tools, which have been based on historic genomic annotations from model organisms, impact our understanding of novel genomes and metagenomes. This hinders the discovery of new genomic information as it results in predictions being biased towards existing knowledge. To date, users have lacked a systematic and replicable approach to identify the strengths and weaknesses of any CoDing Sequence (CDS) prediction tool and allow them to choose the right tool for their analysis. Results We present an evaluation framework (ORForise) based on a comprehensive set of 12 primary and 60 secondary metrics that facilitate the assessment of the performance of CDS prediction tools. This makes it possible to identify which performs better for specific use-cases. We use this to assess 15 ab initio and model-based tools representing those most widely used (historically and currently) to generate the knowledge in genomic databases. We find that the performance of any tool is dependent on the genome being analysed, and no individual tool ranked as the most accurate across all genomes or metrics analysed. Even the top-ranked tools produced conflicting gene collections which could not be resolved by aggregation. The ORForise evaluation framework provides users with a replicable, data-led approach to make informed tool choices for novel genome annotations and for refining historical annotations. Availability https://github.com/NickJD/ORForise Supplementary information Supplementary data are available at Bioinformatics online.

StORF-Reporter: Finding Genes between Genes

“…The concept of a StORF shows that despite all the advancements in computational biology, there are still clear problems in how genome annotation is undertaken. Previous findings (8) confirmed that genome annotation methods in use today such as Prodigal (19) accurately detect the majority of CDS genes (in genomes which use the ‘universal’ codon table). However, in general, while all these tools overpredicted the number of genes, there were still large (>=10-20%) portions of the genomes studied without any annotation.…”

“…The ORForise platform reported that Prodigal was able to identify the vast majority of Ensembl genes from each of the MOs, except for M. genitalium (8). The genomic regions containing no CDS predictions were extracted with UR-Extractor and StORFs were reported from these URs using StORF-Finder.…”

“…Different validation methods were required to determine what was being detected by the StORF sequences. For the StORFs identified from within the URs of the Prodigal annotations, an extension to the ‘ORForise’ platform (8) was developed, which originally was designed to compare annotations from different CDS prediction tools to a reference annotation. This extension, ‘StORForise’, was used to identify the Ensembl CDS genes missed by Prodigal that StORF-Reporter was able to recover.…”

“…To elucidate these regions and overcome the weaknesses and biases in genome annotation, many of which continue to be outlined in the literature (8), we have developed a novel methodology to supplement genome annotation. This approach, named ‘StORF-Reporter’, is performed in two stages.…”

KEMET – A python tool for KEGG Module evaluation and microbial genome annotation expansion

“…Functional annotations of predicted genes were performed using eggNOG-mapper v2 [7] with default parameters. While in principle alternative gene predictions can impact the subsequent functional annotation, previous empirical investigations found negligible performance variation among different tools [23] , [24] . For this reason, our tests focused on benchmarking functional annotation prediction by using a single state-of-the-art gene prediction tool.…”