Mining big data using parsimonious factor, machine learning, variable selection and shrinkage methods

Kim, Hyun Hak; Swanson, Norman R.

doi:10.1016/j.ijforecast.2016.02.012

Cited by 85 publications

(45 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The choice of regularization parameters can impact on the predictive performance of models specified using these sorts of methods. For a discussion further of the regularization parameters, including values to use thereof, please refer to Kim and Swanson (), as well as the papers cited in Kim and Swanson where the various estimation algorithms for these methods are developed.…”

Section: Dimension Reduction and Penalized Regressionmentioning

confidence: 99%

Big data analytics in economics: What have we learned so far, and where should we go from here?

Swanson

Xiong

2018

Canadian J of Economics

Self Cite

View full text Add to dashboard Cite

Research into predictive accuracy testing remains at the forefront of the forecasting field. One reason for this is that rankings of predictive accuracy across alternative models, which under misspecification are loss function dependent, are universally utilized to assess the usefulness of econometric models. A second reason, which corresponds to the objective of this paper, is that researchers are currently focusing considerable attention on so-called big data and on new (and old) tools that are available for the analysis of this data. One of the objectives in this field is the assessment of whether big data leads to improvement in forecast accuracy. In this survey paper, we discuss some of the latest (and most interesting) methods currently available for analyzing and utilizing big data when the objective is improved prediction. Our discussion includes a summary of various so-called dimension reduction, shrinkage and machine learning methods as well as a summary of recent tools that are useful for ranking prediction models associated with the implementation of these methods. We also provide a brief empirical illustration of big data in action, in which we show that big data are indeed useful when predicting the term structure of interest rates.Résumé. L'analyse des données massives en économie : ce qu'on a appris jusqu'à maintenant et quelles directions pour la prochaine étape? Les travaux sur les tests de précision des prévisions demeurent au premier plan dans le monde de la prévision. Une première raison est que les ordonnancements de la précision des prévisions des divers modèles (qui selon le degré de mis-spécification dépend de la fonction de perte) sont universellement utilisés pour calibrer l'utilité des modèles économétriques. Une seconde raison, qui correspond à l'objectif de ce mémoire, est que les chercheurs concentrent une portion considérable de leur attention sur ce qu'on appelle les données massives, et les outils (anciens et nouveaux) disponibles pour analyser ce type de données. Un des objectifs dans ce champ d'études est d'établir si les données massives mènent à participants of the 2017 Deutsche Bundesbank Workshop on Forecasting for useful comments and suggestions.l'amélioration dans la précision des prévisions. Dans cette étude-synthèse, les auteurs examinent quelques-unes des méthodes les plus récentes et les plus intéressantes qui sont disponibles pour analyser et utiliser ce genre de données quand l'objectif est d'améliorer les prévisions. La discussion inclut une présentation succincte d'approches en termes de réduction de dimensions, de rétrécissement, et méthodes d'apprentissage machine, ainsi qu'un résumé succinct d'outillages récents qui sont utiles pour ordonnancer les modèles de prévision associés à la mise en application de ces méthodes. On fournit aussi une brève illustration de données massives en action, dans laquelle on montre que l'analyse des données massives est utile pour prévoir la structure par échéance des taux d'intérêt.JEL classification: C12, C22, C534 In their setu...

show abstract

Section: Dimension Reduction and Penalized Regressionmentioning

confidence: 99%

Big data analytics in economics: What have we learned so far, and where should we go from here?

Swanson

Xiong

2018

Canadian J of Economics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recursive and standard principal component analysis (RPCA and OPCA). PCA is widely used to estimate factors or diffusion indices in large data environments (see Kim & Swanson, (2016), and references cited therein).…”

Section: Estimating Diffusion Indexesmentioning

confidence: 99%

“…• Bridge equation with exogenous variables (BEX). This method is identical to the above CBADL model, except that the model in Step 2 is replaced with 4 Stock and Watson (2002) and Kim and Swanson (2016) implement a version of this model.…”

Section: Benchmark Models and Experimental Setupmentioning

confidence: 99%

See 1 more Smart Citation

Methods for backcasting, nowcasting and forecasting using factor‐MIDAS: With an application to Korean GDP

Kim

Swanson

2017

Journal of Forecasting

Self Cite

View full text Add to dashboard Cite

We utilize mixed‐frequency factor‐MIDAS models for the purpose of carrying out backcasting, nowcasting, and forecasting experiments using real‐time data. We also introduce a new real‐time Korean GDP dataset, which is the focus of our experiments. The methodology that we utilize involves first estimating common latent factors (i.e., diffusion indices) from 190 monthly macroeconomic and financial series using various estimation strategies. These factors are then included, along with standard variables measured at multiple different frequencies, in various factor‐MIDAS prediction models. Our key empirical findings as follows. (i) When using real‐time data, factor‐MIDAS prediction models outperform various linear benchmark models. Interestingly, the “MSFE‐best” MIDAS models contain no autoregressive (AR) lag terms when backcasting and nowcasting. AR terms only begin to play a role in “true” forecasting contexts. (ii) Models that utilize only one or two factors are “MSFE‐best” at all forecasting horizons, but not at any backcasting and nowcasting horizons. In these latter contexts, much more heavily parametrized models with many factors are preferred. (iii) Real‐time data are crucial for forecasting Korean gross domestic product, and the use of “first available” versus “most recent” data “strongly” affects model selection and performance. (iv) Recursively estimated models are almost always “MSFE‐best,” and models estimated using autoregressive interpolation dominate those estimated using other interpolation methods. (v) Factors estimated using recursive principal component estimation methods have more predictive content than those estimated using a variety of other (more sophisticated) approaches. This result is particularly prevalent for our “MSFE‐best” factor‐MIDAS models, across virtually all forecast horizons, estimation schemes, and data vintages that are analyzed.

show abstract

“…On this point, we can consider the approach discussed in this work and pre-select some predictors based on their similarities to the predictand. We refer the readers to a parallel work by Kim and Swanson (2016) for further reading.…”

Section: Appendix B Exploratory Analysis On Distributions Of Clearnementioning

confidence: 99%

Analyzing big time series data in solar engineering using features and PCA

et al. 2017

View full text Add to dashboard Cite

In solar engineering, we encounter big time series data such as the satellite-derived irradiance data and string-level measurements from a utility-scale photovoltaic (PV) system. While storing and hosting big data are certainly possible using today's data storage technology, it is challenging to effectively and efficiently visualize and analyze the data. We consider a data analytics algorithm to mitigate some of these challenges in this work. The algorithm computes a set of generic and/or application-specific features to characterize the time series, and subsequently uses principal component analysis to project these features onto a two-dimensional space. As each time series can be represented by features, it can be treated as a single data point in the feature space, allowing many operations to become more amenable. Three applications are discussed within the overall framework, namely (1) the PV system type identification, (2) monitoring network design, and (3) anomalous string detection. The proposed framework can be easily translated to many other solar engineer applications.

show abstract

Mining big data using parsimonious factor, machine learning, variable selection and shrinkage methods

Cited by 85 publications

References 67 publications

Big data analytics in economics: What have we learned so far, and where should we go from here?

Big data analytics in economics: What have we learned so far, and where should we go from here?

Methods for backcasting, nowcasting and forecasting using factor‐MIDAS: With an application to Korean GDP

Analyzing big time series data in solar engineering using features and PCA

Contact Info

Product

Resources

About