How Many Imputations are Really Needed? Some Practical Clarifications of Multiple Imputation Theory

Abstract

Multiple imputation (MI) and full information maximum likelihood (FIML) are the two most common approaches to missing data analysis. In theory, MI and FIML are equivalent when identical models are tested using the same variables, and when m, the number of imputations performed with MI, approaches infinity. However, it is important to know how many imputations are necessary before MI and FIML are sufficiently equivalent in ways that are important to prevention scientists. MI theory suggests that small values of m, even on the order of three to five imputations, yield excellent results. Previous guidelines for sufficient m are based on relative efficiency, which involves the fraction of missing information (γ) for the parameter being estimated, and m. In the present study, we used a Monte Carlo simulation to test MI models across several scenarios in which γ and m were varied. Standard errors and p-values for the regression coefficient of interest varied as a function of m, but not at the same rate as relative efficiency. Most importantly, statistical power for small effect sizes diminished as m became smaller, and the rate of this power falloff was much greater than predicted by changes in relative efficiency. Based our findings, we recommend that researchers using MI should perform many more imputations than previously considered sufficient. These recommendations are based on γ, and take into consideration one's tolerance for a preventable power falloff (compared to FIML) due to using too few imputations.

How financial strain affects health: Evidence from the Dutch National Bank Household Survey

“…Previously observed data on variables in the analysis make the MAR assumption more realistic (Newsom et al, 2013) and were included along 5 with current demographics in the imputation model. In order to yield sufficient statistical power, fifty imputations were carried out (Graham et al, 2007;Enders, 2010). Feng et al (2013) provide an accessible overview of methodological approaches for dealing with missing data in longitudinal studies.…”

Cost-effectiveness of focal psychodynamic therapy and enhanced cognitive–behavioural therapy in out-patients with anorexia nervosa

“…Intervention group was included as a covariate. To minimize the power-falloff, given the fraction of missing information, 40 imputations were created (Graham et al 2007).…”

A simple and general method for simultaneously accounting for phylogenetic and species sampling uncertainty via Rubin’s rules in comparative analysis

“…Rubin's (1987) initial recommendation of m was low (3-10) probably due to computational limitation at that time, but current thinking is to use much larger m, aiming at over 99% relative efficiency (e.g. Graham et al 2007, von Hippel 2009, Nakagawa 2015. As you see in Equation (13), we obtain a relative efficiency value (ε i ) for every parameter and such values vary among parameters.…”