Gipfeli - High Speed Compression Algorithm

Lenhardt, Rastislav; Alakuijala, Jyrki

doi:10.1109/dcc.2012.19

Cited by 17 publications

(13 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This was also experimentally confirmed on real data gathered from network traffic [9]. Note that a combination of memory-assisted compression and parallel compression techniques that achieve high compression rate as well as high compression speed make compression-based redundancy elimination feasible on high rate links as well [14], [15].…”

Section: Introductionsupporting

confidence: 59%

Wireless Network Compression Via Memory-Enabled Overhearing Helpers

Beirami

Sardari

Fekri

2016

IEEE Trans. Wireless Commun.

View full text Add to dashboard Cite

Traces derived from real-world traffic show that significant redundancy exists at the packet level in mobile network traffic. This has inspired new solutions to suppress the redundancy present in the packet data to manage the explosive traffic. In this paper, we propose a novel approach to perform redundancy elimination by employing universal compression using memory-enabled overhearing helpers without backhaul connectivity, referred to as wireless network compression. The helpers overhear the data packets previously sent by the wireless gateway to various mobile clients within their coverage and use them as side information to reduce the overall communication cost. We study wireless network compression via overhearing helpers from an information-theoretic point of view and conclude that this approach potentially offers a three-fold benefit: 1) offloading the wireless gateway and hence increasing the maximum number of mobile nodes the gateway can reliably serve; 2) reducing the average packet delay; and 3) improving the overall throughput in the network.

show abstract

Section: Introductionsupporting

confidence: 59%

Wireless Network Compression Via Memory-Enabled Overhearing Helpers

Beirami

Sardari

Fekri

2016

IEEE Trans. Wireless Commun.

View full text Add to dashboard Cite

show abstract

“…The main goal in the design of such high-speed algorithms has been to adapt LZ77 compression to achieve highest possible speed and through this process compression performance is traded for speed. As such, the compression performance of high-speed algorithms suffers, for example, compression of the first 1GB of the English Wikipedia using Snappy [42], and Gipfeli [41] has resulted in 530MB (in 2.8sec) and 410MB (in 4.3sec), respectively. However, the implementation of gzip that we experimented on would compress the same input to 320MB (in 41.7sec).…”

Section: Compression Complexitymentioning

confidence: 99%

Packet-Level Network Compression: Realization and Scaling of the Network-Wide Benefits

Beirami

Sardari

Fekri

2016

IEEE/ACM Trans. Networking

View full text Add to dashboard Cite

Abstract-The existence of considerable amount of redundancy in the Internet traffic at the packet level has stimulated the deployment of packet-level redundancy elimination techniques within the network by enabling network nodes to memorize data packets. Redundancy elimination results in traffic reduction which in turn improves the efficiency of network links. In this paper, the concept of network compression is introduced that aspires to exploit the statistical correlation beyond removing large duplicate strings from the flow to better suppress redundancy.In the first part of the paper, we introduce "memory-assisted compression," which utilizes the memorized content within the network to learn the statistics of the information source generating the packets which can then be used toward reducing the length of codewords describing the packets emitted by the source. Using simulations on data gathered from real network traces, we show that memory-assisted compression can result in significant traffic reduction.In the second part of the paper, we study the scaling of the average network-wide benefits of memory-assisted compression. We discuss routing and memory placement problems in network for the reduction of overall traffic. We derive a closed-form expression for the scaling of the gain in Erdős-Rényi random network graphs, where obtain a threshold value for the number of memories deployed in a random graph beyond which networkwide benefits start to shine. Finally, the network-wide benefits are studied on Internet-like scale-free networks. We show that nonvanishing network compression gain is obtained even when only a tiny fraction of the total number of nodes in the network are memory-enabled.

show abstract

“…Several techniques are available to improve the throughput, such as hardware acceleration [3], algorithmic approximations, and computer architecture optimizations [4][5][6][7]. Although these acceleration, approximation, and optimization techniques may accelerate compression, there are many systems where these do not suffice either due to limited speed up or poor compression quality.…”

mentioning

confidence: 99%

A Universal Parallel Two-Pass MDL Context Tree Compression Algorithm

Krishnan

Baron

2015

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

Computing problems that handle large amounts of data necessitate the use of lossless data compression for efficient storage and transmission. We present a novel lossless universal data compression algorithm that uses parallel computational units to increase the throughput. The length-N input sequence is partitioned into B blocks. Processing each block independently of the other blocks can accelerate the computation by a factor of B, but degrades the compression quality. Instead, our approach is to first estimate the minimum description length (MDL) context tree source underlying the entire input, and then encode each of the B blocks in parallel based on the MDL source. With this two-pass approach, the compression loss incurred by using more parallel units is insignificant. Our algorithm is work-efficient, i.e., its computational complexity is O(N/B). Its redundancy is approximately B log(N/B) bits above Rissanen's lower bound on universal compression performance, with respect to any context tree source whose maximal depth is at most log(N/B). We improve the compression by using different quantizers for states of the context tree based on the number of symbols corresponding to those states. Numerical results from a prototype implementation suggest that our algorithm offers a better trade-off between compression and throughput than competing universal data compression algorithms.

show abstract

Gipfeli - High Speed Compression Algorithm

Cited by 17 publications

References 7 publications

Wireless Network Compression Via Memory-Enabled Overhearing Helpers

Wireless Network Compression Via Memory-Enabled Overhearing Helpers

Packet-Level Network Compression: Realization and Scaling of the Network-Wide Benefits

A Universal Parallel Two-Pass MDL Context Tree Compression Algorithm

Contact Info

Product

Resources

About