Rafiki

Mahgoub, Ashraf; Wood, Paul; Ganesh, Sachandhan; Mitra, Subrata; Gerlach, Wolfgang; Harrison, Travis; Meyer, Folker; Grama, Ananth; Bagchi, Saurabh; Chaterji, Somali

doi:10.1145/3135974.3135991

Cited by 37 publications

(13 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, given that finding the optimal configuration traverses a non-convex trajectory, we have used simulated annealing to reduce the chances of getting stuck in a local minima. Further, some EC tools may have performance-sensitive configuration parameters that may have interdependencies and, in such cases, a surrogate model that can encode dependencies, as in our recent work [62,63], can be used. Finally, we have compared Lerna 's findings with ground truth results on Illumina short reads and PacBio and Nanopore long reads.…”

Section: Discussionmentioning

confidence: 99%

Lerna: transformer architectures for configuring error correction tools for short- and long-read genome sequencing

et al. 2022

Self Cite

View full text Add to dashboard Cite

Background Sequencing technologies are prone to errors, making error correction (EC) necessary for downstream applications. EC tools need to be manually configured for optimal performance. We find that the optimal parameters (e.g., k-mer size) are both tool- and dataset-dependent. Moreover, evaluating the performance (i.e., Alignment-rate or Gain) of a given tool usually relies on a reference genome, but quality reference genomes are not always available. We introduce Lerna for the automated configuration of k-mer-based EC tools. Lerna first creates a language model (LM) of the uncorrected genomic reads, and then, based on this LM, calculates a metric called the perplexity metric to evaluate the corrected reads for different parameter choices. Next, it finds the one that produces the highest alignment rate without using a reference genome. The fundamental intuition of our approach is that the perplexity metric is inversely correlated with the quality of the assembly after error correction. Therefore, Lerna leverages the perplexity metric for automated tuning of k-mer sizes without needing a reference genome. Results First, we show that the best k-mer value can vary for different datasets, even for the same EC tool. This motivates our design that automates k-mer size selection without using a reference genome. Second, we show the gains of our LM using its component attention-based transformers. We show the model’s estimation of the perplexity metric before and after error correction. The lower the perplexity after correction, the better the k-mer size. We also show that the alignment rate and assembly quality computed for the corrected reads are strongly negatively correlated with the perplexity, enabling the automated selection of k-mer values for better error correction, and hence, improved assembly quality. We validate our approach on both short and long reads. Additionally, we show that our attention-based models have significant runtime improvement for the entire pipeline—18$$\times$$ × faster than previous works, due to parallelizing the attention mechanism and the use of JIT compilation for GPU inferencing. Conclusion Lerna improves de novo genome assembly by optimizing EC tools. Our code is made available in a public repository at: https://github.com/icanforce/lerna-genomics.

show abstract

Section: Discussionmentioning

confidence: 99%

Lerna: transformer architectures for configuring error correction tools for short- and long-read genome sequencing

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Considering the different selection ratios and time costs, we set two TCs in our experiments, i.e., 2h and 3h. Furthermore, it is worth mentioning that in real production systems, especially in big data scenarios, the time cost of collecting a performance measurement varies from minutes to hours (Babu et al, 2009;Bei et al, 2015;Mahgoub et al, 2017). Since the time cost of manual labeling by human experts is relatively stable, a larger time cost for performance measurements would bring more potential for CM-CASL.…”

Section: Experimental Methodologymentioning

confidence: 99%

“…These approaches apply regression techniques to model the relationship. The most commonly used models include CART (Guo et al, 2013(Guo et al, , 2018Nair et al, 2018a;Sarkar et al, 2015;Valov et al, 2015;Nair et al, 2017), RF (Valov et al, 2015;Bei et al, 2015), neural networks (Ha and Zhang, 2019;Mahgoub et al, 2017;Zheng et al, 2014), Gaussian process regression (Duan et al, 2009;Thummala and Babu, 2010;Van Aken et al, 2017;Zhang et al, 2018), and Support Vector Regression (SVR) (Chen et al, 2019;Valov et al, 2015).…”

Section: Related Workmentioning

confidence: 99%

“…In many previous studies, performance modeling is formulated as a regression problem, and different regression-based models (e.g., neural network (Ha and Zhang, 2019;Mahgoub et al, 2017;Zheng et al, 2014), Classification And Regression Trees (CART) (Guo et al, 2013(Guo et al, , 2018Nair et al, 2018a;Sarkar et al, 2015;Valov et al, 2015;Nair et al, 2017), Random Forest (RF) (Valov et al, 2015;Bei et al, 2015), etc.) are trained with a limited set of labeled samples from the configuration space.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CM-CASL: Comparison-based Performance Modeling of Software Systems via Collaborative Active and Semisupervised Learning

Cao¹,

Li²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Configuration tuning for large software systems is generally challenging due to the complex configuration space and expensive performance evaluation. Most existing approaches follow a two-phase process, first learning a regressionbased performance prediction model on available samples and then searching for the configurations with satisfactory performance using the learned model. Such regression-based models often suffer from the scarcity of samples due to the enormous time and resources required to run a large software system with a specific configuration. Moreover, previous studies have shown that even a highly accurate regression-based model may fail to discern the relative merit between two configurations, whereas performance comparison is actually one fundamental strategy for configuration tuning. To address these issues, this paper proposes CM-CASL, a Comparison-based performance Modeling approach for software systems via Collaborative Active and Semisupervised Learning. CM-CASL learns a classification model that compares the performance of two given configurations, and enhances the samples through a collaborative labeling process by both human experts and classifiers using an integration of active and semisupervised learning. Experimental results demonstrate that CM-CASL outperforms two state-of-the-art performance modeling approaches in terms of both classification accuracy and rank accuracy, and thus provides a better performance model for the subsequent work of configuration tuning.

show abstract

“…One common method to measure the performance of a database is to run some benchmarking tool for an extended amount of time or performing a high given number of operations, in order to factor out the stochasticity in the evaluation and obtain reliable measurements [11,32,44]. On the other hand, selecting the best database configuration for a certain task requires hundreds or thousands of evaluations, and therefore we strive for an evaluation to be as fast as possible [25,27].…”

Section: Introductionmentioning

confidence: 99%

Determining a consistent experimental setup for benchmarking and optimizing databases

Silva-Muñoz

Calderón

Franzin

et al. 2021

Proceedings of the Genetic and Evolutionary Computation Conference Companion

View full text Add to dashboard Cite

The evaluation of the performance of an IT system is a fundamental operation in its benchmarking and optimization. However, despite the general consensus on the importance of this task, little guidance is usually provided to practitioners who need to benchmark their IT system. In particular, many works in the area of database optimization do not provide an adequate amount of information on the setup used in their experiments and analyses. In this work we report an experimental procedure that, through a sequence of experiments, analyzes the impact of various choices in the design of a database benchmark, leading to the individuation of an experimental setup that balances the consistency of the results with the time needed to obtain them. We show that the minimal experimental setup we obtain is representative also of heavier scenarios, which make it possible for the results of optimization tasks to scale.

show abstract

Rafiki

Cited by 37 publications

References 34 publications

Lerna: transformer architectures for configuring error correction tools for short- and long-read genome sequencing

Lerna: transformer architectures for configuring error correction tools for short- and long-read genome sequencing

CM-CASL: Comparison-based Performance Modeling of Software Systems via Collaborative Active and Semisupervised Learning

Determining a consistent experimental setup for benchmarking and optimizing databases

Contact Info

Product

Resources

About