Two-Level Data Compression using Machine Learning in Time Series Database

Yu, Xinyang; Peng, Yanqing; Li, Feifei; Wang, Sheng; Shen, Xiaowei; Mai, Huijun; Xie, Yue

doi:10.1109/icde48307.2020.00119

Cited by 22 publications

(10 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In time-series data, the pattern can change over time and a fixed compression scheme may not work well for the entire duration. Yu et al [55] propose a two-level compression framework, where a scheme space is constructed by extracting global features at the top level and a compression schema is selected for each point at the bottom level. The proposed AMMMO framework incorporates compression primitives and the control parameters, which define the compression scheme space.…”

Section: Data Compressionmentioning

confidence: 99%

A Survey on Deep Reinforcement Learning for Data Processing and Analytics

Cai

Cui

Xiong

et al. 2022

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Data processing and analytics are fundamental and pervasive. Algorithms play a vital role in data processing and analytics where many algorithm designs have incorporated heuristics and general rules from human knowledge and experience to improve their effectiveness. Recently, reinforcement learning, deep reinforcement learning (DRL) in particular, is increasingly explored and exploited in many areas because it can learn better strategies in complicated environments it is interacting with than statically designed algorithms. Motivated by this trend, we provide a comprehensive review of recent works focusing on utilizing DRL to improve data processing and analytics. First, we present an introduction to key concepts, theories, and methods in DRL. Next, we discuss DRL deployment on database systems, facilitating data processing and analytics in various aspects, including data organization, scheduling, tuning, and indexing. Then, we survey the application of DRL in data processing and analytics, ranging from data preparation, natural language processing to healthcare, fintech, etc. Finally, we discuss important open challenges and future research directions of using DRL in data processing and analytics.

show abstract

Section: Data Compressionmentioning

confidence: 99%

A Survey on Deep Reinforcement Learning for Data Processing and Analytics

Cai

Cui

Xiong

et al. 2022

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…The study in [18] proposes an algorithm which combines a recurrent neural network predictor and a lossless compression method; and uses genomic and text datasets and achieves around 20% reduction over the traditional compression method, i.e., Gzip. The work in [19] presents a two-level approach that selects a compression framework for individual data points in time-series and then proposes a neural network structure that tunes parameter values automatically. As a result, the framework is capable of improving compression ratio by up to 120% compared to other traditional compression methods.…”

Section: State Of the Artmentioning

confidence: 99%

“…The study in [18] uses genomic and text datasets and combines a recurrent neural network predictor and lossless compression method; and achieves around 20% reduction over Gzip on the datasets. The work in [19] uses a reinforcement learning-based approach for compression of time-series data and improves compression ratio by up to 120% (with an average of 50%), compared to compression methods called Gorilla, MO (Middle-Out), and Snappy. Then, the work studies bandwidth utilization of the method using a combined CPU and GPU platform for Gorilla, MO, and Snappy.…”

Section: F Comparison With the State-of-the-artmentioning

confidence: 99%

Quality of Monitoring for Cellular Networks

Motlagh¹,

Kapoor²,

Alhalaseh³

et al. 2022

IEEE Trans. Netw. Serv. Manage.

View full text Add to dashboard Cite

5G networks and beyond introduce a larger number of Network Elements (NEs) and functions than former cellular generations. The increase in NEs will, thus, result in significantly increasing the Management-Plane (M-Plane) data collected from the NEs. Therefore, the conventional centralized Network Management Systems (NMSs) will face fundamental challenges in processing the M-Plane data. In this paper, we present the concept of Quality of Monitoring (QoM) as a solution, which is able to reduce the M-Plane data already at the NEs. First, QoM aggregates the raw M-Plane data into Key Performance Indicators (KPIs). To these KPIs, the QoM applies a datadriven algorithm to define information loss limits for QoM classes specific for each KPI time series. Then, the QoM applies the classes for compressing the KPI data utilizing a lossycompression method, which is a derivative of the Piece-Wise Constant Approximation (PWCA) algorithm. To evaluate the performance of the QoM solution, we use M-Plane raw data from a live LTE network and calculate four KPIs, while each KPI has different statistical characteristics. We also define three QoM classes named Exact, Optimized, and Sharp. For all KPIs, the class Optimized has a higher compression rate than the class Exact, while the class Sharp has the highest compression rate. Assuming that, for example, NEs of a network produce 280 MB of raw data containing information that needs to be transferred to the network operations center; we use KPIs to represent the information contents of the data, and QoM solution to transfer the data over the network. As a result, the QoM solution achieves an estimated 95% compression gain from the raw data in transfer.

show abstract

“…Therefore, according to [ 39 ], there are many universal compression techniques, such as (static or adaptive) Huffman and arithmetic coding, which are ubiquitous in real-world applications. In addition, it is known that they can be categorized into two types: lossy compression and lossless compression.…”

Section: Data Compressionmentioning

confidence: 99%

“…In the study [ 39 ], the author proposed a two-level compression model that selects a proper compression scheme for each individual point, making it possible for diverse patterns to be captured with fine granularity. Based on this model, they designed and implemented the Adaptive Multi-Model Middle-Out (AMMMO) framework, which provides access to a set of control parameters to categorize data patterns.…”

Section: Related Workmentioning

confidence: 99%

An Evolving TinyML Compression Algorithm for IoT Environments Based on Data Eccentricity

Signoretti

Silva

Andrade

et al. 2021

Sensors

View full text Add to dashboard Cite

Currently, the applications of the Internet of Things (IoT) generate a large amount of sensor data at a very high pace, making it a challenge to collect and store the data. This scenario brings about the need for effective data compression algorithms to make the data manageable among tiny and battery-powered devices and, more importantly, shareable across the network. Additionally, considering that, very often, wireless communications (e.g., low-power wide-area networks) are adopted to connect field devices, user payload compression can also provide benefits derived from better spectrum usage, which in turn can result in advantages for high-density application scenarios. As a result of this increase in the number of connected devices, a new concept has emerged, called TinyML. It enables the use of machine learning on tiny, computationally restrained devices. This allows intelligent devices to analyze and interpret data locally and in real time. Therefore, this work presents a new data compression solution (algorithm) for the IoT that leverages the TinyML perspective. The new approach is called the Tiny Anomaly Compressor (TAC) and is based on data eccentricity. TAC does not require previously established mathematical models or any assumptions about the underlying data distribution. In order to test the effectiveness of the proposed solution and validate it, a comparative analysis was performed on two real-world datasets with two other algorithms from the literature (namely Swing Door Trending (SDT) and the Discrete Cosine Transform (DCT)). It was found that the TAC algorithm showed promising results, achieving a maximum compression rate of 98.33%. Additionally, it also surpassed the two other models regarding the compression error and peak signal-to-noise ratio in all cases.

show abstract

Two-Level Data Compression using Machine Learning in Time Series Database

Cited by 22 publications

References 17 publications

A Survey on Deep Reinforcement Learning for Data Processing and Analytics

A Survey on Deep Reinforcement Learning for Data Processing and Analytics

Quality of Monitoring for Cellular Networks

An Evolving TinyML Compression Algorithm for IoT Environments Based on Data Eccentricity

Contact Info

Product

Resources

About