The Sense and Non-Sense of Holdout Sample Validation in the Presence of Endogeneity

Abstract

Market response models based on field-generated data need to address potential endogeneity in the regressors to obtain consistent parameter estimates. Another requirement is that market response models predict well in a holdout sample. With both requirements combined, it may seem reasonable to subject an endogeneity-corrected model to a holdout prediction task, and this is quite common in the academic marketing literature. One may be inclined to expect that the consistent parameter estimates obtained via instrumental variables (IV) estimation predict better than the biased ordinary least squares (OLS) estimates. This paper shows that this expectation is incorrect. That is, if the holdout sample is similar to the estimation sample so that the regressors are endogenous in both samples, holdout sample validation favors regression estimates that are not corrected for endogeneity (i.e., OLS) over estimates that are corrected for endogeneity (i.e., IV estimation). We also discuss ways in which holdout samples may be used sensibly in the presence of endogeneity. A key takeaway is that if consistent parameter estimates are the primary model objective, the model should be validated with an exogenous (rather than endogenous) holdout sample.

The Effectiveness of Field Price Discretion: Empirical Evidence from Auto Lending

“…12 Endogeneity correction is further crucial for such a counterfactual analysis. Ebbes et al (2011) state that it is necessary to correct for endogeneity when the goal is a better explanation of the customer behavior in a different environment instead of better predictions in the same environment. any dealer discretion).…”

The Challenge of Retaining Customers Acquired with Free Trials

“…To assess model performance, we evaluate the retention model's in-sample fit and out-of-sample fit (cross-sectional and longitudinal) on three fit measures: hit probability (Gilbride, Allenby, and Brazell 2006), top-decile lift, and Gini coefficient (Lemmens and Croux 2006). While in-and out-of-sample fit are not suited to compare models with and without endogeneity correction, they can be used to validate different approaches that all correct for endogeneity (Ebbes, Papies, and Van Heerde 2011). Table 3 shows that model M1 based on matched samples and including both copula and Mundlak terms outperforms the other models for seven out of nine fit criteria.…”

Optimizable and implementable aggregate response modeling for marketing decision support

“…The goodness-of-fit criterion does not represent a meaningful metric because it must be weaker if we replace an endogenous variable with its instrumented variable. Ebbes et al (2011) investigate whether it is possible to assess the suitability of instrumental variables with holdout validation. They find this to be impossible because holdout validity is always better for regressors of biased OLS than it is for corrected OLS with IV.…”