Selection and sorting with limited storage

Munro, J. Ian; Paterson, Mike

doi:10.1109/sfcs.1978.32

Cited by 139 publications

(200 citation statements)

References 1 publication

Supporting

Mentioning

200

Contrasting

Order By: Relevance

“…For discussion simplification, we assume that a data element has a single value and the total order means an increasing order of data values. Rank-Element (RE) Query: Given a rank r, find the element with rank r. It was shown [20] that any algorithm for computing exact φ-quantiles of an ordered set of N data elements requires Ω(N 1/κ ) space if κ scans of the data set are allowed. Consider that quantiles computation may be immediately transformed to RE queries.…”

Section: Problem Statementmentioning

confidence: 99%

“…Among various statistics, order statistics computation is one of the most challenging, and is employed in many real applications, such as web ranking aggregation and log mining [1,12], sensor data analysis [14], trends and fleeting opportunities detection in stock markets [3,23], and load balanced data partitioning for distributed computation [25,27]. Most order statistics computation problems require memory size linearly proportional to the size of a data stream for exact answers by one-scan techniques [1,20]; this may be impractical in data stream applications where streams are massive in size and fast in arrival speed. Consequently, approximate computation is a good alternative.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Space-efficient Relative Error Order Sketch over Data Streams

Zhang

Lin

et al. 2006

22nd International Conference on Data Engineering (ICDE'06)

View full text Add to dashboard Cite

We consider the problem of continuously maintaining order sketches over data streams with a relative rank error guarantee . Novel space-efficient and one-scan randomised techniques are developed. Our first randomised algorithm can guarantee such a relative error precision with confidence 1 − δ using O( 1 2 log 1 δ log 2 N ) space, where N is the number of data elements seen so far in a data stream. Then, a new one-scan space compression technique is developed. Combined with the first randomised algorithm, the one-scan space compression technique yields another one-scan randomised algorithm that guarantees the space requirement is O( 1 log( 1 log 1 δ ) log 2+α N 1−1/2 α ) (for α > 0) on average while the worst case space remains O( 1 2 log 1 δ log 2 N ). These results are immediately applicable to approximately computing quantiles over data streams with a relative error guarantee and significantly improve the previous best space bound O( 1 3 log 1 δ log N ). Our extensive experiment results demonstrate that both techniques can support an on-line computation against high speed data streams.

show abstract

Section: Problem Statementmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Space-efficient Relative Error Order Sketch over Data Streams

Zhang

Lin

et al. 2006

22nd International Conference on Data Engineering (ICDE'06)

View full text Add to dashboard Cite

show abstract

“…For example, a one-pass algorithm to compute approximate quantiles has been given by Munro and Paterson [20]. Utilizing the knowledge on the data distribution, Alsabti, et al, have presented an algorithm to estimate the quantile values with provable error bounds [21].…”

Section: Introductionmentioning

confidence: 99%

“…Utilizing the knowledge on the data distribution, Alsabti, et al, have presented an algorithm to estimate the quantile values with provable error bounds [21]. A distribution independent quantile estimation [22] can also be done in one pass with error guarantees that costs less memory than [20], [21]. …”

Section: Introductionmentioning

confidence: 99%

A two-pass exact algorithm for selection on Parallel Disk Systems

Tian

Rajasekaran

2013

2013 IEEE Symposium on Computers and Communications (ISCC)

View full text Add to dashboard Cite

Numerous OLAP queries process selection operations of “top N”, median, “top 5%”, in data warehousing applications. Selection is a well-studied problem that has numerous applications in the management of data and databases since, typically, any complex data query can be reduced to a series of basic operations such as sorting and selection. The parallel selection has also become an important fundamental operation, especially after parallel databases were introduced. In this paper, we present a deterministic algorithm Recursive Sampling Selection (RSS) to solve the exact out-of-core selection problem, which we show needs no more than (2 + ε) passes (ε being a very small fraction). We have compared our RSS algorithm with two other algorithms in the literature, namely, the Deterministic Sampling Selection and QuickSelect on the Parallel Disks Systems. Our analysis shows that DSS is a (2 + ε)-pass algorithm when the total number of input elements N is a polynomial in the memory size M (i.e., N = Mc for some constant c). While, our proposed algorithm RSS runs in (2 + ε) passes without any assumptions. Experimental results indicate that both RSS and DSS outperform QuickSelect on the Parallel Disks Systems. Especially, the proposed algorithm RSS is more scalable and robust to handle big data when the input size is far greater than the core memory size, including the case of N ≫ Mc.

show abstract

“…The study of upper bounds on the time-space trade-off for sorting was initiated by Munro and Paterson [11] who gave a time-space focused algorithm in a model with tapeinput. For algorithms with random access input the first fully scalable, and till now the best upper bound for the time-space trade-off was given by Frederickson [8], who showed that for ÐÓ Ò Ë Ḉ Òµ there exists a comparison RAM algorithm sorting Ò keys in time Ì and space Ë such that Ì ¡ Ë Ḉ Ò ¾ ÐÓ Òµ.…”

Section: Related Workmentioning

confidence: 99%

Optimal time-space trade-offs for sorting

Pagter

Rauhe

Proceedings 39th Annual Symposium on Foundations of Computer Science (Cat. No.98CB36280)

View full text Add to dashboard Cite

We study the fundamental problem of sorting in a sequential model of computation and in particular consider the time-space trade-off (product of time and space) for this problem.Beame has shown a lower bound of ª´Ò ¾ µ for this product leaving a gap of a logarithmic factor up to the previously best known upper bound of Ç´Ò ¾ ÐÓ Òµ due to Frederickson. Since then, no progress has been made towards tightening this gap.The main contribution of this paper is a comparison based sorting algorithm which closes the gap by meeting the lower bound of Beame. The time-space product Ç´Ò ¾ µ upper bound holds for the full range of space bounds between ÐÓ Ò and Ò ÐÓ Ò. Hence in this range our algorithm is optimal for comparison based models as well as for the very powerful general models considered by Beame.

show abstract

Selection and sorting with limited storage

Abstract: Abstnact. When selecting fnom, on sorting, a file stored on a reao-only tape and the irrternal storage is nather linritedrseveral !'asses o-f the inpr-rt tape may be requir,ed. h-e strrdy the relation befween t}'re arnount of internal stor"age availabl-e and the number of

Cited by 139 publications

References 1 publication

Space-efficient Relative Error Order Sketch over Data Streams

Space-efficient Relative Error Order Sketch over Data Streams

A two-pass exact algorithm for selection on Parallel Disk Systems

Optimal time-space trade-offs for sorting

Contact Info

Product

Resources

About