Demand Estimation with Machine Learning and Model Combination

Abstract

The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research. NBER working papers are circulated for discussion and comment purposes. They have not been peerreviewed or been subject to the review by the NBER Board of Directors that accompanies official NBER publications.

The Ensemble Method for Censored Demand Prediction

“…Machine learning methods have better predictive properties than traditional econometric approaches -this fact has been repeatedly proven in a number of papers. For example, [2] compare neural networks and multinomial logit models in brands' shares prediction and find that neural networks predicts better; [33] shows that regression trees works comparatively better than logistic regression for a larger dataset and also demonstrates some advantages of such methods as bagging, bootstrapping and boosting over traditional econometric approaches; and, finally, [5] compares the predictive power of a number of traditional econometric models and methods of machine learning in a within product category prediction problem, and comes to an unambiguous conclusion of the better performance of the latter.…”

“…And if the task of RMSE minimizing has been solving in computer science for quite a long time, the application of the developed models to economic problems with the subsequent possibility of an adequate results interpretation has become widespread recently. To date, there are some studies that partially fill the gap between traditional econometric approach and ML methods in the context of demand prediction ( [15], [33], [6], [5], [34], [30]). In the one of the most recent development [5] authors consider several machine learning techniques and compare them with traditional econometric models, empirically proving better predictive power of the formers.…”

“…To date, there are some studies that partially fill the gap between traditional econometric approach and ML methods in the context of demand prediction ( [15], [33], [6], [5], [34], [30]). In the one of the most recent development [5] authors consider several machine learning techniques and compare them with traditional econometric models, empirically proving better predictive power of the formers. Further, in order to improve the out-of sample prediction accuracy they develop a method of the underlying models combination via constrained linear regression.…”

“…Further, in order to improve the out-of sample prediction accuracy they develop a method of the underlying models combination via constrained linear regression. In our study, we generalize the algorithm described in the [5] by for the case of censored dependent data. Our motivation in combining algorithms of machine learning and censorship estimation is encouraged by the possibility to increase the predictive accuracy of well known models.…”

“…But with the increased availability of scanner data that contains individual data on purchases, machine learning (ML) methods turns out to outperform econometric models in a demand prediction problem. Methods of machine learning allowed to obtain more precise out-of-sample predictions on large datasets and take into account unobserved consumers' heterogeneity and other nonregularities in sales data ( [2], [6], [5], [33]). Furthermore, ML methods demonstrate a higher convergence rates compared to non-parametric econometric models which led to the prevalence of their use in cases with a large number of possible predictors.…”