Statistical Hypothesis Testing for Assessing Monte Carlo Estimators: Applications to Image Synthesis

Abstract

Image synthesis algorithms are commonly compared on the basis of running times and/or perceived quality of the generated images. In the case of Monte Carlo techniques, assessment often entails a qualitative impression of convergence toward a reference standard and severity of visible noise; these amount to subjective assessments of the mean and variance of the estimators, respectively. In this paper we argue that such assessments should be augmented by well-known statistical hypothesis testing methods. In particular, we show how to perform a number of such tests to assess random variables that commonly arise in image synthesis such as those estimating irradiance, radiance, pixel color, etc. We explore five broad categories of tests: 1) determining whether the mean is equal to a reference standard, such as an analytical value, 2) determining that the variance is bounded by a given constant, 3) comparing the means of two different random variables, 4) comparing the variances of two different random variables, and 5) verifying that two random variables stem from the same parent distribution. The level of significance of these tests can be controlled by a parameter. We demonstrate that these tests can be used for objective evaluation of Monte Carlo estimators to support claims of zero or small bias and to provide quantitative assessments of variance reduction techniques. We also show how these tests can be used to detect errors in sampling or in computing the density of an importance function in MC integrations.

Fourier analysis of stochastic sampling strategies for assessing bias and variance in integration

“…For example, statistics such as Ripley's K and L statistics [Ripley 1977] are commonly used to model point distributions, while tools such as the variogram and autocorrelation are used for assessing distributions. In computer graphics, methods such as frequency domain analysis [Dippe and Wold 1985;Cook 1986;Mitchell 1991], a measure of spatial discrepancy [Shirley 1991], statistical tests of hypothesis [Subr and Arvo 2007], and point correlation [Öztireli and Gross 2012] have been Illustration of errors in reconstruction and integration introduced by the sampling spectrum. Only the amplitude spectra are shown, for illustration.…”

Fourier Analysis of Correlated Monte Carlo Importance Sampling

“…Error analysis of integrators The error due to point distributions [Rip77, IPSS08] have been well studied in the statistical literature [OZ00, Owe13, Hes03]. In computer graphics, various measures such as spatial discrepancy [Shi91], statistical tests of hypothesis [SA07] and pair correlations [Ö16] have been proposed for quantifying the error characteristics of sampling‐based estimators. Since correlations impact the spectral distribution, many researchers [Coo86, DW85, Mit91, Lem09] have analysed these approaches in the Fourier domain.…”

Sampling strategies

“…In this section we present a simple and compact algorithm that allows generation of stratified samples [102]according to a linearly-varying density function over a triangle with vertices A, B, C and vertex weights w a , w b and w c . A, B, C denote the position vectors of the vertices.…”