The use of asymmetric numeral systems as an accurate replacement for Huffman coding

Duda, Jarek; Tahboub, Khalid; Gadgil, Neeraj J.; Delp, Edward J.

doi:10.1109/pcs.2015.7170048

Cited by 92 publications

(55 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The SAM to BAM conversion requires relatively less computation since it uses gzip along with some preprocessing. The version of Scramble used in the experiments uses the range-coder variant of the Asymmetric Numerical System, which has been shown to be faster than arithmetic encoding [16]. Our compression and decompression times are middle-of-the-pack.…”

Section: Resultsmentioning

confidence: 99%

GeneComp, a New Reference-Based Compressor for SAM Files

Long

Hernáez

Ochoa

et al. 2017

2017 Data Compression Conference (DCC)

View full text Add to dashboard Cite

The affordability of DNA sequencing has led to unprecedented volumes of genomic data. These data must be stored, processed, and analyzed. The most popular format for genomic data is the SAM format, which contains information such as alignment, quality values, etc. These files are large (on the order of terabytes), which necessitates compression. In this work we propose a new reference-based compressor for SAM files, which can accommodate different levels of compression, based on the specific needs of the user. In particular, the proposed compressor GeneComp allows the user to perform lossy compression of the quality scores, which have been proven to occupy more than half of the compressed file (when losslessly compressed). We show that the proposed compressor GeneComp overall achieves better compression ratios than previously proposed algorithms when working on lossless mode.

show abstract

Section: Resultsmentioning

confidence: 99%

GeneComp, a New Reference-Based Compressor for SAM Files

Long

Hernáez

Ochoa

et al. 2017

2017 Data Compression Conference (DCC)

View full text Add to dashboard Cite

show abstract

“…The modern lossless compression is mainly based on two approaches to entropy coding: Huffman coding (HC) and arithmetic coding (AC) [18]. Later, the Asymmetric numeral systems approach was introduced with a high performance of combining the fast speed of Huffman with the high compression rate of arithmetic coding [19].…”

Section: Lossless Compressionmentioning

confidence: 99%

Experimental Study on Machine Learning with Approximation to Data Streams

Zhang

Jiang

2019

2019 Sixth International Conference on Internet of Things: Systems, Management and Security (IOTSMS)

View full text Add to dashboard Cite

Realtidsöverföring av dataströmmar möjliggör många dataanalyser och maskininlärningsapplikationer inom områdena t.ex. massiv IoT och industriell automatisering. Stor datavolym för dessa strömmar är en betydande börda eller omkostnad inte bara för transportnätet utan också för motsvarande applikationsservrar. Därför fokuserar forskare och forskare om att minska mängden data som behövs för att överföras via datakomprimeringar och approximationer. Datakomprimeringstekniker som förlustkomprimering kan minska datavolymen betydligt med priset för datainformation. Samtidigt är datakomprimering mycket beroende av motsvarande applikationer. Men när du använder dekomprimerade data i en viss dataanalysapplikation som maskininlärning, kan resultaten påverkas på grund av informationsförlusten. I denna artikel gjorde författaren en studie om effekterna av datakomprimering på maskininlärningsapplikationerna.

show abstract

“…For example, zStandard also makes use a of a faster and more effective method for evaluating entropy, known as Asymmetric Numeral System (ANS), than classic compression algorithms. ANS significantly improves compression ratio in data with large degrees of randomness (Duda, 2013, andDuda et al, 2015).…”

Section: Blocked Compressionmentioning

confidence: 99%

MRCZ – A proposed fast compressed MRC file format and direct detector normalization strategies

McLeod

Righetto

Stewart

2017

Preprint

View full text Add to dashboard Cite

The introduction of high-speed CMOS detectors is fast marching the field of transmission electron microscopy into an intersection with the computer science field of big data. Automated data pipelines to control the instrument and the initial processing steps are imposing more and more onerous requirements on data transfer and archiving. We present a proposal for expansion of the venerable MRC file format to combine integer decimation and lossless compression to reduce storage requirements and improve file read/write times by >1000 % compared to uncompressed floating-point data. The integer decimation of data necessitates application of the gain normalization and outlier pixel removal at the data destination, rather than the source. With direct electron detectors, the normalization step is typically provided by the vendor and is not opensource. We provide robustly tested normalization algorithms that perform at-least as well as vendor software. We show that the generation of hot pixels is a highly dynamic process in direct electron detectors, and that outlier pixels must be detected on a stack-by-stack basis. In comparison, the low-frequency bias features of the detectors induced by the electronics on-top of the active layer, are extremely stable with time. Therefore we introduce a stochastic-based approach to identify outlier pixels and smoothly filter them, such that the degree of correlated noise in micrograph stacks is reduced. Both a priori and a posteriori gain normalization approaches that are compatible with pipeline image processing are discussed. The a priori approach adds a gammacorrection to the gain reference, and the a posteriori approach normalized by a moving average of time-adjacent stacks, with the current stack being knocked-out, known as the KOMA (knock-out moving average) filter. The combination of outlier filter and KOMA normalization over ~25 frames can reduce the correlated noise in movies to nearly zero. Sample libraries and a command-line utility are hosted at github.com/em-MRCZ and released under the BSD license.

show abstract

The use of asymmetric numeral systems as an accurate replacement for Huffman coding

Cited by 92 publications

References 4 publications

GeneComp, a New Reference-Based Compressor for SAM Files

GeneComp, a New Reference-Based Compressor for SAM Files

Experimental Study on Machine Learning with Approximation to Data Streams

MRCZ – A proposed fast compressed MRC file format and direct detector normalization strategies

Contact Info

Product

Resources

About