Noise and Bias In Square-Root Compression Schemes

Abstract

We investigate data compression schemes for proposed all-sky diffraction-limited visible/NIR sky surveys aimed at the dark energy problem. We show that lossy squareroot compression to 1 bit of noise per pixel, followed by standard lossless compression algorithms, reduces the images to 2.5-4 bits per pixel, depending primarily upon the level of cosmic-ray contamination of the images. Compression to this level adds noise equivalent to ≤ 10% penalty in observing time. We derive an analytic correction to flux biases inherent to the square-root compression scheme. Numerical tests on simple galaxy models confirm that galaxy fluxes and shapes are measured with systematic biases 10 −4 induced by the compression scheme, well below the requirements of supernova and weak gravitational lensing dark-energy experiments. An accompanying paper (Vanderveld et al. 2009) bounds the shape biases using realistic simulated images of the high-Galactic-latitude sky. The square-root preprocessing step has advantages over simple (linear) decimation when there are many bright objects or cosmic rays in the field, or when the background level will vary.

What Bandwidth Do I Need for My Image?

“…techniques (such as gzip) than do the floating-point originals (Gaztañaga et al 2001;Watson 2002;White & Greenfield 1999;Pence et al 2009;Bernstein et al 2009). In the Δ ¼ 0:5σ representation, after lossless compression, storage and transmission of the image "costs" only a few bits per noisedominated pixel.…”

Lossy Compression of Weak-Lensing Data

“…We use the compression scheme, including bias correction, as described in Bernstein et al (2010). We provide a brief description here.…”

“…The codec process has a similar effect on the data to that of read noise in the readout electronics or Poisson statistics. Bernstein et al (2010) refine the basic square-root codec in equation 2with choices for A, B, and C, which maintain constant σ=N step at any signal level for given detector gain and read noise; a prescription for slight departures from (2) to produce a codec that has uniform behavior of N step as the signal increases; and a correction to the decompressed values, which eliminates small biases in the mean signal introduced by the codec process. We will primarily focus on an implementation of the square-root compression algorithm that yields b ¼ 1, which we naively expect to provide the best compromise between our desires for a high compression level but for low image degradation, but we will also do some tests with a coarser b ¼ 0:71 and a finer b ¼ 1:41 level of compression.…”

“…The lossy compression algorithms used in this article can all be implemented as simple lookup tables following gain g and digitization of the analog detector output. Using the notation of Bernstein et al (2010), these three codecs are constructed as follows. Output code i is assigned to all input integers in the range N i AE ðΔ i À 1Þ=2, where Δ i is an integer giving N step for this output code.…”

“…We will also employ two other codec schemes at times: (1) b ¼ 1:41, for which Δ 0 i follow the sequence f0; 1; 0; 1; 0; 1; …g, and (2) b ¼ 0:71, which has the same lookup table as the original codec but with g ¼ 1 e ADU À1 before digitization. See Bernstein et al (2010) for a complete description of the compression algorithm and the exact correction factors for decompression.…”