CoTS: A Scalable Framework for Parallelizing Frequency Counting over Data Streams

Das, Sudipto; Antony, Shyam; Agrawal, Divyakant; Abbadi, Amr El

doi:10.1109/icde.2009.231

Cited by 10 publications

(17 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the figure, x-axis shows a pair of adjacent windows compared to one another, and y-axis shows the calculated ∆ for the associated pair of windows (normalized). As shown in the figure, pairs of windows of (1,2), (4,5), (5,6), (9,10), (12,13) and (15,16) show relatively high ∆'s, whereas windows of (3,4), (6,7), (8,9), (10,11) and (13,14) reported minor changes, agreeing in overall with the results of clustered patterns. Although not adjacent, two windows of 1 and 4 look alike in Figure 4, and its calculated degree of changes is fairly low with ∆ (1,4) = 0.34 (not shown in Figure 5).…”

Section: The Proposed Approachsupporting

confidence: 79%

“…As in Figure 3, the figure shows clustered pattern changes over time with discernible similarity/dissimilarity between windows. We can see that several time windows such as (3,4), (6,7), (8,9) and (10,11) look pretty similar. On the other hand, some others such as (1,2), (2,3), (12,13) and (15,16) show highly different patterns.…”

Section: The Proposed Approachmentioning

confidence: 96%

“…Network monitoring could use streaming data mining techniques, and sampling methods and data reduction techniques were studied by frequency counting [7], histogram [11], sliding windows [8], random sampling [22], wavelets [20], and dimensionality reduction [24]. Many of these sampling methods provide a quick understanding of the monitored data stream, but characterizing accurate data distribution from the streaming data is still a challenge, especially with the recent hardware advances, which produces data records at a much higher rate.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Peeking Network States with Clustered Patterns

Kim

Sim

2015

View full text Add to dashboard Cite

Network traffic monitoring has long been a core element for effective network management and security. However, it is still a challenging task with a high degree of complexity for comprehensive analysis when considering multiple variables and ever-increasing traffic volumes to monitor. For example, one of the widely considered approaches is to scrutinize probabilistic distributions, but it poses a scalability concern and multivariate analysis is not generally supported due to the exponential increase of the complexity. In this work, we propose a novel method for network traffic monitoring based on clustering, one of the powerful deep-learning techniques. We show that the new approach enables us to recognize clustered results as patterns representing the network states, which can then be utilized to evaluate "similarity" of network states over time. In addition, we define a new quantitative measure for the similarity between two compared network states observed in different time windows, as a supportive means for intuitive analysis. Finally, we demonstrate the clustering-based network monitoring with public traffic traces, and show that the proposed approach using the clustering method has a great opportunity for feasible, costeffective network monitoring.

show abstract

Section: The Proposed Approachsupporting

confidence: 79%

Section: The Proposed Approachmentioning

confidence: 96%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Peeking Network States with Clustered Patterns

Kim

Sim

2015

View full text Add to dashboard Cite

show abstract

“…Additionally, the proposed design paradigm allows replacing locks by much cheaper atomic synchronization primitives. In a previous work [8], we demonstrate the effectiveness of the "thread cooperation" paradigm for intra-operator parallelism of stream operations, and in this paper, we use this paradigm for multi-stream analysis. Concepts similar to "thread cooperation", such as the escrow transactional model [26], have earlier been used in the field of databases.…”

Section: Multi-stream Analysis and Multicoresmentioning

confidence: 93%

Thread cooperation in multicore architectures for frequency counting over multiple data streams

et al. 2009

Self Cite

View full text Add to dashboard Cite

Many real-world data stream analysis applications such as network monitoring, click stream analysis, and others require combining multiple streams of data arriving from multiple sources. This is referred to as multi-stream analysis. To deal with high stream arrival rates, it is desirable that such systems be capable of supporting very high processing throughput. The advent of multicore processors and powerful servers driven by these processors calls for efficient parallel designs that can effectively utilize the parallelism of the multicores, since performance improvement is possible only through effective parallelism. In this paper, we address the problem of parallelizing multi-stream analysis in the context of multicore processors. Specifically, we concentrate on parallelizing frequent elements, top-k, and frequency counting over multiple streams. We discuss the challenges in designing an efficient parallel system for multi-stream processing. Our evaluation and analysis reveals that traditional "contention" based locking results in excessive overhead and wait, which in turn leads to severe performance degradation in modern multicore architectures. Based on our analysis, we propose a "cooperation" based locking paradigm for efficient parallelization of frequency counting. The proposed "cooperation" based paradigm removes waits associated with synchronization, and allows replacing locks by much cheaper atomic synchronization primitives. Our implementation of the proposed paradigm to parallelize a well known frequency counting algorithm shows the benefits of the proposed "cooperation" based locking paradigm when compared to the traditional "contention" based locking paradigm. In our experiments, the proposed "cooperation" based design outperforms the traditional "contention" based design by a factor of 2 − 5.5X for synthetic zipfian data sets.

show abstract

“…The major disadvantage is lacking scalability, as it only outperforms the ''contention" based design by a factor of 2-5.5X. Since the ''contention" based design underperforms the single thread design [19], the speedup of ''cooperation" based design compared to the single thread design will be less than 2-5.5 correspondingly. That is to say, adding more threads in the ''cooperation" based design will not lead to more speedup.…”

Section: Related Workmentioning

confidence: 99%