Correct and stable sorting for overflow streaming data with a limited storage size and a uniprocessor

Abstract

Tremendous quantities of numeric data have been generated as streams in various cyber ecosystems. Sorting is one of the most fundamental operations to gain knowledge from data. However, due to size restrictions of data storage which includes storage inside and outside CPU with respect to the massive streaming data sources, data can obviously overflow the storage. Consequently, all classic sorting algorithms of the past are incapable of obtaining a correct sorted sequence because data to be sorted cannot be totally stored in the data storage. This paper proposes a new sorting algorithm called streaming data sort for streaming data on a uniprocessor constrained by a limited storage size and the correctness of the sorted order. Data continuously flow into the storage as consecutive chunks with chunk sizes less than the storage size. A theoretical analysis of the space bound and the time complexity is provided. The sorting time complexity is O (n), where n is the number of incoming data. The space complexity is O (M), where M is the storage size. The experimental results show that streaming data sort can handle a million permuted data by using a storage whose size is set as low as 35% of the data size. This proposed concept can be practically applied to various applications in different fields where the data always overflow the working storage and sorting process is needed.

Fast continuous streaming sort in big streaming data environment under fixed-size single storage

“…The basic studied problem in this paper focuses on the sorting of streaming data under the constraints of limited working memory size and a uniprocessor similar to the problem studied in [ 17 ]. Although the sorting algorithm in [ 34 ] can efficiently sort the streaming data under the imposed constraints, it assumes that there are no duplicate data in the streaming input. This assumption is impractical for many real-world data sets.…”

“…Since this paper concerns the improvement of the algorithm in [ 34 ], the definitions of introduced sequence types must be recapped here as follows.…”

“…Recently, Chaikhan et al [ 34 ] proposed an algorithm called streaming data sort to continuously sort incoming big streaming data by using a uniprocessor and only one internal fixed-size working memory. The memory size is much smaller than the total streaming data size.…”

“…Streaming data also possess this lifecycle trait without exception. The method of streaming data sort [ 34 ] is incapable of managing expired data after being sorted and captured in a compact representation. However, this issue is resolved in this study.…”

“…Streaming data sort [ 34 ] proposed a sorting algorithm for large streaming data by representing sorted sub-sequences in forms of compact groups so that the sorting process could be executed under a limited working memory size on a uniprocessor. The compacting process is a major concept of streaming data sort to efficiently represent the sorted data set to avoid memory overflow.…”