Characteristics, such as the support limit and distortions, of minimum mean-squared error (MSE) N-level uniform and nonuniform scalar quantizers are studied for the family of the generalized gamma density functions as N increases. For the study, MSE-optimal scalar quantizers are designed at integer rates from 1 to 16 bits/sample, and their characteristics are compared with corresponding asymptotic formulas. The results show that the support limit formulas are generally accurate. They also show that the distortion of nonuniform quantizers is observed to converge to the Panter-Dite asymptotic constant, whereas the distortion of uniform quantizers exhibits slow or even stagnant convergence to its corresponding Hui-Neuhoff asymptotic constant at the studied rate range, though it may stay at a close proximity to the asymptotic constant for the Rayleigh and Laplacian pdfs.Additional terms in the asymptote result in quite considerable accuracy improvement, making the formulas useful especially when rate is 8 or greater.

BSTRACTThe paper studies characteristics of the minimum mean-square error symmetric scalar quantizers for the generalized gamma, Bucklew-Gallagher and Hui-Neuhoff probability density functions. Toward this goal, asymptotic formulas for the inner-and outermost thresholds, and distortion are derived herein for nonuniform quantizers for the Bucklew-Gallagher and Hui-Neuhoff densities, parallelling the previous studies for the generalized gamma density, and optimal uniform and nonuniform quantizers are designed numerically and their characteristics tabulated for integer rates up to 20 and 16 bits, respectively, except for the Hui-Neuhoff density.The assessed asymptotic formulas are found consistently more accurate as the rate increases, essentially making their asymptotic convergence to true values numerically acceptable at the studied bit range, except for the Hui-Neuhoff density, in which case they are still consistent and suggestive of convergence. Also investigated is the uniqueness problem of the differentiation method for finding optimal step sizes of uniform quantizers: it is observed that, for the commonly studied densities, the distortion has a unique local minimizer, hence showing that the differentiation method yields the optimal step size, but also observed that it leads to multiple solutions to numerous generalized gamma densities.Key Words : scalar quantization, generalized gamma source, Bucklew-Gallagher source, Hui-Neuhoff source, asymptotic formulas, MSE distortion Ⅰ. IntroductionQuantization is an integral part of converting an analog signal to a digital form for further processing, communication and/or storage, such as in a GPS system [1] , a soft-decision error correction system [2] or a multiple antenna mobile system [3] . In such applications it is important to estimate the performance and key characteristics of quantizers for the system design and performance analysis. In general without actual design it is difficult to estimate accurately thresholds, quantization levels, and the mean-square error (MSE) distortion of an optimal (minimum MSE) quantizer. However, when the number of levels is large, the high resolution quantization theory [4][5][6][7][8][9][10] has discovered approximation formulas that are increasingly accurate with more number of levels for some of such characteristics. This paper extends in particular these two studies [9][10] , respectively, to nonuniform quantizers and to source probability density functions (pdfs) with heavier tails; investigates the uniqueness problem of the differentiation approach [9] for finding optimal step sizes; and assesses accuracies of these formulas by comparing them with those of numerically designed quantizers.

No abstract