What Bandwidth Do I Need for My Image?

Price-Whelan, Adrian M.

doi:10.1086/651009

Cited by 4 publications

(1 citation statement)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the context of the petascale astronomy era, prior work considered questions such as: which data format/model should be used to enable work with Big Data (e.g. Kitaeff et al, 2012;Kitaeff et al, 2014, this issue;Natusch, 2014;Mink et al, 2014;Price et al, 2014); what bandwidth to keep in case of lossy compression (Price-Whelan & Hogg, 2010); and whether lossy compression affects analysis (e.g. White & Percival, 1994;Shamir & Nemiroff, 2005;Pence et al, 2010;Vohl, 2013;Peters & Kitaeff, 2014).…”

Section: Big Data Data Format and Data Compressionmentioning

confidence: 99%

Data compression in the petascale astronomy era: A GERLUMPH case study

Vohl

Fluke

Vernardos

2015

Astronomy and Computing

View full text Add to dashboard Cite

As the volume of data grows, astronomers are increasingly faced with choices on what data to keep -and what to throw away. Recent work evaluating the JPEG2000 (ISO/IEC 15444) standards as a future data format standard in astronomy has shown promising results on observational data. However, there is still a need to evaluate its potential on other type of astronomical data, such as from numerical simulations. GERLUMPH (the GPU-Enabled High Resolution cosmological MicroLensing parameter survey) represents an example of a data intensive project in theoretical astrophysics. In the next phase of processing, the ≈ 27 terabyte GERLUMPH dataset is set to grow by a factor of 100 -well beyond the current storage capabilities of the supercomputing facility on which it resides. In order to minimise bandwidth usage, file transfer time, and storage space, this work evaluates several data compression techniques. Specifically, we investigate off-the-shelf and custom lossless compression algorithms as well as the lossy JPEG2000 compression format. Results of lossless compression algorithms on GERLUMPH data products show small compression ratios (1.35:1 to 4.69:1 of input file size) varying with the nature of the input data. Our results suggest that JPEG2000 could be suitable for other numerical datasets stored as gridded data or volumetric data. When approaching lossy data compression, one should keep in mind the intended purposes of the data to be compressed, and evaluate the effect of the loss on future analysis. In our case study, lossy compression and a high compression ratio do not significantly compromise the intended use of the data for constraining quasar source profiles from cosmological microlensing.

show abstract

Section: Big Data Data Format and Data Compressionmentioning

confidence: 99%

Data compression in the petascale astronomy era: A GERLUMPH case study

Vohl

Fluke

Vernardos

2015

Astronomy and Computing

View full text Add to dashboard Cite

show abstract

Optimal Compression of Floating-Point Astronomical Images Without Significant Loss of Information

Pence¹,

White²,

Seaman³

2010

PUBL ASTRON SOC PAC

View full text Add to dashboard Cite

ABSTRACT. We describe a compression method for floating-point astronomical images that gives compression ratios of 6-10 while still preserving the scientifically important information in the image. The pixel values are first preprocessed by quantizing them into scaled integer intensity levels, which removes some of the uncompressible noise in the image. The integers are then losslessly compressed using the fast and efficient Rice algorithm and stored in a portable FITS format file. Quantizing an image more coarsely gives greater image compression, but it also increases the noise and degrades the precision of the photometric and astrometric measurements in the quantized image. Dithering the pixel values during the quantization process can greatly improve the precision of measurements in the images. This is especially important if the analysis algorithm relies on the mode or the median, which would be similarly quantized if the pixel values are not dithered. We perform a series of experiments on both synthetic and real astronomical CCD images to quantitatively demonstrate that the magnitudes and positions of stars in the quantized images can be measured with the predicted amount of precision. In order to encourage wider use of these image compression methods, we have made available a pair of general-purpose image compression programs, called fpack and funpack, which can be used to compress any FITS format image.

show abstract

Improved Background Subtraction for the Sloan Digital Sky Survey Images

Blanton

Kazin

Muna

et al. 2011

The Astronomical Journal

550

520

View full text Add to dashboard Cite

We describe a procedure for background subtracting Sloan Digital Sky Survey (SDSS) imaging that improves the resulting detection and photometry of large galaxies on the sky. Within each SDSS drift scan run, we mask out detected sources and then fit a smooth function to the variation of the sky background. This procedure has been applied to all SDSS-III Data Release 8 images, and the results are available as part of that data set. We have tested the effect of our background subtraction on the photometry of large galaxies by inserting fake galaxies into the raw pixels, reanalyzing the data, and measuring them after background subtraction. Our technique results in no size-dependent bias in galaxy fluxes up to half-light radii r 50 ∼ 100 arcsec; in contrast, for galaxies of that size the standard SDSS photometric catalog underestimates fluxes by about 1.5 mag. Our results represent a substantial improvement over the standard SDSS catalog results and should form the basis of any analysis of nearby galaxies using the SDSS imaging data.

show abstract

What Bandwidth Do I Need for My Image?

Cited by 4 publications

References 5 publications

Data compression in the petascale astronomy era: A GERLUMPH case study

Data compression in the petascale astronomy era: A GERLUMPH case study

Optimal Compression of Floating-Point Astronomical Images Without Significant Loss of Information

Improved Background Subtraction for the Sloan Digital Sky Survey Images

Contact Info

Product

Resources

About