Comparing Different Early Warning Systems: Results from a Horse Race Competition Among Members of the Macro-Prudential Research Network

Abstract

Over the recent decades researchers in academia and central banks have developed early warning systems (EWS) designed to warn policy makers of potential future economic and financial crises. These EWS are based on diverse approaches and empirical models. In this paper we compare the performance of nine distinct models for predicting banking crises resulting from the work of the Macroprudential Research Network (MaRs) initiated by the European System of Central Banks. In order to ensure comparability, all models use the same database of crises created by MaRs and comparable sets of potential early warning indicators. We evaluate the models' relative usefulness by comparing the ratios of false alarms and missed crises and discuss implications for pratical use and future research. We find that multivariate models, in their many appearances, have great potential added value over simple signalling models. One of the main policy recommendations coming from this exercise is that policy makers can benefit from taking a broad methodological approach when they develop models to set macro-prudential instruments.The obligatory copyright note: We certify that we have the right to deposit the contribution with MPRA.

Comparing logit-based early warning systems: Does the duration of systemic banking crises matter?

“…Specifically, our empirical analysis rests on the use of low frequency, yearly data. At least in samples of advanced economies, recent papers have developed EWSs based on the binomial logit model using quarterly measures of systemic distress (see Alessi et al, 2015, for a review of the literature). Compared to yearly data, quarterly observations would allow for a more refined analysis of the role played by the duration of crisis in driving our conclusions, an analysis that is in our agenda.…”

“…Demirgüç-Kunt and Detragiache (2000), Davis and Karim (2008a;2008b) and Alessi et al, (2015) show that crisis probabilities estimated through the binomial multivariate logit exhibit lower type I (missed crises) and type II (false alarms) errors than the signals approach and therefore provide a more accurate basis for building an EWS.…”

Toward Robust Early-Warning Models: A Horse Race, Ensembles and Model Uncertainty

“…3 Receiver operating characteristics (ROC) curves and the area under the ROC curve (AUC) are also used for comparing performance of early-warning models and indicators. The ROC curve plots, for the complete range of τ ∈ [0, 1], the conditional probability of positives to the conditional probability of negatives:…”

“…First, there are few objective and thorough comparisons of conventional and novel methods, and thus neither unanimity on an overall ranking of methods nor on a single best-performing method. Though the horse race conducted among members of the Macro-prudential Research Network of the European System of Central Banks aims at a prediction competition, it does not provide a solid basis for objective performance comparisons [3]. Even though disseminating information of models underlying discretionary policy discussion is a valuable task, the panel of presented methods are built and applied in varying contexts.…”

Systemic banking crisis early warning systems using dynamic Bayesian networks

“…In addition, various other static classifiers have been applied, which include classification trees, neural networks, and random forests ( Alessi et al, 2015 ). Dynamic methods are scarce in EWS literature.…”