Engineering an External Memory Minimum Spanning Tree Algorithm

Dementiev, Roman; Sanders, Peter; Schultes, Dominik; Sibeyn, Jop F.

doi:10.1007/1-4020-8141-3_17

Cited by 40 publications

(49 citation statements)

References 22 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Empirically, we found that the external memory implementation of [16] (EM MST) performs better than the one in [1]. Table 2 shows the total time required for their deterministic preprocessing using CO MST and EM MST on low diameter random graphs and on high diameter line graphs.…”

Section: Computation Modelsmentioning

confidence: 98%

2007 Proceedings of the Ninth Workshop on Algorithm Engineering and Experiments (ALENEX)

Applegate¹,

Brodal²

2007

View full text Add to dashboard Cite

Breadth first search (BFS) traversal on massive graphs in external memory was considered non-viable until recently, because of the large number of I/Os it incurs. Ajwani et al. [3] showed that the randomized variant of the o(n) I/O algorithm of Mehlhorn and Meyer [24] (MM BFS) can compute the BFS level decomposition for large graphs (around a billion edges) in a few hours for small diameter graphs and a few days for large diameter graphs. We improve upon their implementation of this algorithm by reducing the overhead associated with each BFS level, thereby improving the results for large diameter graphs which are more difficult for BFS traversal in external memory. Also, we present the implementation of the deterministic variant of MM BFS and show that in most cases, it outperforms the randomized variant. The running time for BFS traversal is further improved with a heuristic that preserves the worst case guarantees of MM BFS. Together, they reduce the time for BFS on large diameter graphs from days shown in [3] to hours. In particular, on line graphs with random layout on disks, our implementation of the deterministic variant of MM BFS with the proposed heuristic is more than 75 times faster than the previous best result for the randomized variant of MM BFS in [3].

show abstract

Section: Computation Modelsmentioning

confidence: 98%

2007 Proceedings of the Ninth Workshop on Algorithm Engineering and Experiments (ALENEX)

Applegate¹,

Brodal²

2007

View full text Add to dashboard Cite

show abstract

“…Another implementation of a parallel approach is reported in [42] with calculations distributed between processors and with the use of MPI. Using different types of parallel random access memory (PRAM) model, Dementiev et al found [37] a good solution for minimizing the time of communication using external memory when constructing an MST, while in [6], authors dealt with the construction of a spanning tree using Symmetric Multiprocessors, though there is no guarantee that it will find an MST.…”

Section: Parallelizationmentioning

confidence: 99%

Scaling up data mining algorithms: review and taxonomy

García-Pedrajas

Haro-García

2012

Prog Artif Intell

View full text Add to dashboard Cite

The overwhelming amount of data that are now available in any field of research poses new problems for data mining and knowledge discovery methods. Due to this huge amount of data, most of the current data mining algorithms are inapplicable to many real-world problems. Data mining algorithms become ineffective when the problem size becomes very large. In many cases, the demands of the algorithm in terms of the running time are very large, and mining methods cannot be applied when the problem grows. This aspect is closely related to the time complexity of the method. A second problem is linked with performance; although the method might be applicable, the size of the search space prevents an efficient execution, and the resulting solutions are unsatisfactory. Two approaches have been used to deal with this problem: scaling up data mining algorithms and data reduction. However, because data reduction is a data mining task itself, this technique also suffers from scalability problems. Thus, for many problems, especially when dealing with very large datasets, the only way to deal with the aforementioned problems is to scale up the data mining algorithm. Many efforts have been made to obtain methods that can be used to scale up existing data mining algorithms. In this paper, we review the methods that have been used to address the problem of scalability. We focus on general ideas, rather than specific implementations, that can be used to provide a general view of the current approaches for scaling up data mining methods. A taxonomy of the algorithms is proposed, and many examples of different tasks are presented. Among the different techniques used for data mining, we will pay special attention to evolutionary methods, because these methods have been used very successfully in many data mining tasks.

show abstract

“…However, before [3] there was no actual implementation of an external MST algorithm (or for any other nontrivial external graph problem). The reason was that previous algorithms were complicated to implement and have large constant factors that have never been exposed in the analysis.…”

Section: Designmentioning

confidence: 99%

“…We give examples of challenges and results that are a more or less random sample biased to results we know well. Throughout this paper, we will demonstrate the methodology using the external minimum spanning tree (MST) algorithm from [3] as an example. This example was chosen because it is at the same time simple and illustrates the methodology in most of its aspects.…”

Section: Introductionmentioning

confidence: 99%

Algorithm Engineering – An Attempt at a Definition

Sanders

2009

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. This paper defines algorithm engineering as a general methodology for algorithmic research. The main process in this methodology is a cycle consisting of algorithm design, analysis, implementation and experimental evaluation that resembles Popper's scientific method. Important additional issues are realistic models, algorithm libraries, benchmarks with real-world problem instances, and a strong coupling to applications. Algorithm theory with its process of subsequent modelling, design, and analysis is not a competing approach to algorithmics but an important ingredient of algorithm engineering.

show abstract

Engineering an External Memory Minimum Spanning Tree Algorithm

Cited by 40 publications

References 22 publications

2007 Proceedings of the Ninth Workshop on Algorithm Engineering and Experiments (ALENEX)

2007 Proceedings of the Ninth Workshop on Algorithm Engineering and Experiments (ALENEX)

Scaling up data mining algorithms: review and taxonomy

Algorithm Engineering – An Attempt at a Definition

Contact Info

Product

Resources

About