Demand forecasting with high dimensional data: The case of SKU retail sales forecasting with intra- and inter-category promotional information

Abstract

In marketing analytics applications in OR, the modeler often faces the problem of selecting key variables from a large number of possibilities. For example, SKU level retail store sales are affected by inter and intra category effects which potentially need to be considered when deciding on promotional strategy and producing operational forecasts, but no research has put this well accepted concept into forecasting practice: an obvious obstacle is the ultra-high dimensionality of the variable space. This paper develops a four steps methodological framework to overcome the problem. It is illustrated by investigating the value of both intra-and inter-category SKU level promotional information in improving forecast accuracy. The method consists of the identification of potentially influential categories, the building of the explanatory variable space, variable selection and model estimation by a multistage LASSO regression, and the use of a rolling scheme to generate forecasts. The success of this new method for dealing with high dimensionality is demonstrated by improvements in forecasting accuracy compared to alternative methods of simplifying the variable space. The empirical results show that models integrating more information perform significantly better than the baseline model when using the proposed methodology framework. In general, we can improve the forecasting accuracy by 14.3 percent over the model using only the SKU's own predictors. But of the improvements achieved, 88.1 percent of it comes from the intra-category information, and only 11.9 percent from the inter-category information. The substantive marketing results also have implications for promotional category management.

Estimating Promotion Effects Using Big Data: A Partially Profiled LASSO Model with Endogeneity Correction*

“…The extracted principal components and time dummy variables then serve as the independent variables in a linear regression to forecast the retailer's sales and profits. Though this method has been widely adopted to handle high‐dimensional data in demand forecasting, it has some disadvantages that may compromise its predictive ability (Ma et al., ). For example, the principal components are extracted only from the predictors, and thus the valuable information for predicting the dependent variable may be lost.…”

“…Among the benchmark models, the standard LASSO method has the smallest prediction error for both sales (MSE = 0.0072) and profits (MSE = 0.0122) of the retailer. Ridge regression has relatively large prediction errors (for sales: MSE = 0.0234; for profits: MSE = 0.0188); this is because it cannot perform variable selection, and the inclusion of highly correlated variables attenuates its predictive ability (Ma et al., ). The PCR method has relatively good performance in terms of prediction accuracy, and the random forests method performs even better.…”

“…The positive promotion effects on the retailer's overall performance may be caused by the cross‐category demand effect. For example, sales of products in the salty snacks category are found to significantly increase the sales of products in five other categories (Ma et al., ).…”

“…First, our model controls for the endogeneity of promotion depth in estimating the high‐dimensional promotion effects and advances the application of marketing analytics in data‐rich environments. Although the standard LASSO method was introduced into the statistics literature more than 20 years ago (Tibshirani, ), it has rarely been applied in the research area of marketing (Ma, Fildes, & Huang, ) and few prior studies have considered the endogeneity issue when using the method. Compared with traditional methods, our model can estimate the promotion effects of a large number of products on retailer‐level performance with high efficiency and less endogeneity bias.…”

Estimating Promotion Effects Using Big Data: A Partially Profiled LASSO Model With Endogeneity Correction

“…First, our model controls for the endogeneity of promotion depth in estimating the high-dimensional promotion effects and advances the application of marketing analytics in data-rich environments. Although the standard LASSO method was introduced into the statistics literature more than 20 years ago (Tibshirani, 1996), it has rarely been applied in the research area of marketing (Ma, Fildes, & Huang, 2016) and few prior studies have considered the endogeneity issue when using the method. Compared with traditional methods, our model can estimate the promotion effects of a large number of products on retailer-level performance with high efficiency and less endogeneity bias.…”