Bias-Variance Trade-Off in Continuous Test Norming

Abstract

In continuous test norming, the test score distribution is estimated as a continuous function of predictor(s). A flexible approach for norm estimation is the use of generalized additive models for location, scale, and shape. It is unknown how sensitive their estimates are to model flexibility and sample size. Generally, a flexible model that fits at the population level has smaller bias than its restricted nonfitting version, yet it has larger sampling variability. We investigated how model flexibility relates to bias, variance, and total variability in estimates of normalized z scores under empirically relevant conditions, involving the skew Student t and normal distributions as population distributions. We considered both transversal and longitudinal assumption violations. We found that models with too strict distributional assumptions yield biased estimates, whereas too flexible models yield increased variance. The skew Student t distribution, unlike the Box–Cox Power Exponential distribution, appeared problematic to estimate for normally distributed data. Recommendations for empirical norming practice are provided.

A tutorial on regression-based norming of psychological tests with GAMLSS.

“…It simplifies to the normal distribution when ν = 1 and τ = ∞. In a simulation study (Voncken et al, 2020), it appeared problematic to estimate the reparametrized version of the skew Student t distribution for simulated data sampled from a normal population, presumably because of extremely large estimated τ parameters. The distribution could be estimated well for skewed data.…”

“…The sensitivity of certain GAMLSS norm models (i.e., involving normal, skew Student t , and BCPE distributions) to different forms and sources of model misspecification has been examined in a simulation study (Voncken et al, 2020). This study showed that models with too strict distributional assumptions yield biased estimates, whereas too flexible models yield increased variance.…”

“…For details on the simulation study and their interpretation, we refer to Voncken, Albers, and Timmerman (2020). Here, we provide an illustration of the sensitivity of model estimation to model specification.…”

“…Here, we provide an illustration of the sensitivity of model estimation to model specification. From the simulation study by Voncken et al (2020), we selected a specific condition to highlight effects of estimating the model that fits at the population level, a too strict model, and a too flexible model. Specifically, we drew a sample of N = 1,000 from a normal population model, with a linear relationship between the norm-predictor age and its expected value, and heteroscedasticity (denoted as the Li-HeNo condition in Voncken et al, 2020).…”

“…This tutorial builds on recent developments in GAMLSS for regression-based norming (Oosterhuis, van der Ark, & Sijtsma, 2016; Voncken, Kneib, Albers, Umlauf, & Timmerman, 2020; Voncken, Albers, & Timmerman, 2020; Voncken, Albers, & Timmerman, 2019a; Voncken et al, 2019b) and the experiences we gained in norming various psychological tests (Grob & Hagmann-von Arx, 2018; Grob et al, 2018; Rommelse et al, 2018; Tellegen & Laros, 2017; Voncken, Timmerman, Spikman, & Huitema, 2018).…”