Slipstream: High-Performance Lossless Compression for Streaming Synchronized Waveform Monitoring Data

Blair, Steven M.; Costello, Jason

doi:10.1109/sgsma51733.2022.9805997

Cited by 4 publications

(1 citation statement)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been many studies on caching streaming data on computational nodes before processing. For example, Blair [1] studied a data compression method using wavelet transform to reduce bandwidth requirements and improve end-to-end efficiency and latency for streaming data. An efficient partitioning can decrease the memory overhead of the distributed computational nodes.…”

Section: Introductionmentioning

confidence: 99%

A Partitioning and Distributed Caching Approach Based on Adaptive Spectral Clustering for Big Data Streams

2023

Proceedings of 2023 the 13th International Workshop on Computer Science and Engineering

View full text Add to dashboard Cite

Big data streams with diversity are generally processed by parallel computing environments with multiple computational nodes. Before processing, the big data streams need to be partitioned into sub-streams and cached on each computational node for subsequent processing. Existing partitioning methods are difficult to process streams with diversity and high-dimensional characteristics. Partitioning with low quality leads to unreasonable cache placing, which in turn leads to more data migration, lower computational efficiency, and smaller velocity of big data streams on a processing system. Inspired by the advantages of spectral clustering in identifying arbitrary manifolds, an approach of partitioning for big data streams based on spectral clustering is proposed, which transforms the partitioning of streams received during each micro window into the clustering of similarity graphs. We formulate an optimization problem for data items received during each micro window. Then, we present an algorithm to optimize the similarity graphs. With the characteristics of data streams changing gradually in adjacent windows, a distributed caching algorithm based on stream partitioning is presented for continuous windows. Experimental analysis shows that the proposed method can significantly improve the velocity and efficiency of the system for stream processing.

show abstract

Section: Introductionmentioning

confidence: 99%