Toward Reliable Benchmarking of Solar Flare Forecasting Methods

Abstract

Solar flares occur in complex sunspot groups, but it remains unclear how the probability of producing a flare of a given magnitude relates to the characteristics of the sunspot group. Here, we use Geostationary Operational Environment Satellite X-ray flares and McIntosh group classifications from solar cycles 21 and 22 to calculate average flare rates for each McIntosh class and use these to determine Poisson probabilities for different flare magnitudes. Forecast verification measures are studied to find optimum thresholds to convert Poisson flare probabilities into yes/no predictions of cycle 23 flares. A case is presented to adopt the true skill statistic (TSS) as a standard for forecast comparison over the commonly used Heidke skill score (HSS). In predicting flares over 24 hr, the maximum values of TSS achieved are 0.44 (C-class), 0.53 (M-class), 0.74 (X-class), 0.54 ( M1.0), and 0.46 ( C1.0). The maximum values of HSS are 0.38 (C-class), 0.27 (M-class), 0.14 (X-class), 0.28 ( M1.0), and 0.41 ( C1.0). These show that Poisson probabilities perform comparably to some more complex prediction systems, but the overall inaccuracy highlights the problem with using average values to represent flaring rate distributions.

A Hybrid Supervised/Unsupervised Machine Learning Approach to Solar Flare Prediction

“…Many skill scores can be found in literature for the assessment of flare prediction performances (Bloomfield et al 2012). All these scores are linked to the forecast contingency tables made up of four elements:…”

“…Predicting solar flares requires, first of all, the determination of parameters such as properties of sunspot groups or of the coronal magnetic field configuration, that are thought to be important for the understanding of fundamental processes in solar plasma physics. Second, at a more technological level, these parameters are used as input values for algorithms that realize predictions providing, for example (but not exclusively), a binary flare/no-flare outcome (Bloomfield et al 2012;Wheatland 2004;Gallagher et al 2002).…”

Space Weather Quarterly Volume 14, Issue 2, 2017

“…MOSWOC uses a database containing GOES X-ray flare and McIntosh classifications for this purpose from various sources: Kildhahl data from 1969 to 1976 supplied by Bloomfield et al [2012], SWPC data from 1988 to 1996 supplied by Bloomfield et al [2012], and data from 1996 to 2011 supplied by the ASSA system. This database is used to calculate an average daily flare rate, , for each McIntosh classification (see Bloomfield et al [2012] for more details). The MOSWOC forecaster then calculates the flare probabilities for M-and X-class flares using the Poisson statistics technique of Gallagher et al [2002], where the probability of observing one or more flares in a 24 h period is 1 − e − .…”

“…These percentage probabilities are used as a basis for the flare forecasts issued to end-users by MOSWOC. It is worth noting that the same method is used (as of the date this paper was published) for automatic flare forecasts issued at SolarMonitor.org (http://solarmonitor.org/ forecast.php), except only the Bloomfield et al [2012] data are used to calculate average flare rates.…”

“…This entails deciding a threshold at which the probabilistic values become a "yes/no" forecast and then calculating metrics such as the Heidke skill score and True skill score [Barnes and Leka, 2008;Crown, 2012;Bloomfield et al, 2012]. More recently, however, the community has looked to operational meteorological verification techniques more suitable for probabilistic forecasting, increasingly presenting reliability diagrams and relative operating characteristic (ROC) curves alongside these traditional skill scores [Guerra et al, 2015;Barnes et al, 2016;Cui et al, 2016].…”