A statistical study of the performance of the Hakamada-Akasofu-Fry version 2 numerical model in predicting solar shock arrival times at Earth during different phases of solar cycle 23

Abstract

Abstract. The performance of the Hakamada Akasofu-Fry, version 2 (HAFv.2) numerical model, which provides predictions of solar shock arrival times at Earth, was subjected to a statistical study to investigate those solar/interplanetary circumstances under which the model performed well/poorly during key phases (rise/maximum/decay) of solar cycle 23. In addition to analyzing elements of the overall data set (584 selected events) associated with particular cycle phases, subsets were formed such that those events making up a particular sub-set showed common characteristics. The statistical significance of the results obtained using the various sets/subsets was generally very low and these results were not significant as compared with the hit by chance rate (50 %). This implies a low level of confidence in the predictions of the model with no compelling result encouraging its use. However, the data suggested that the success rates of HAFv.2 were higher when the background solar wind speed at the time of shock initiation was relatively fast. Thus, in scenarios where the background solar wind speed is elevated and the calculated success rate significantly exceeds the rate by chance, the forecasts could provide potential value to the customer. With the composite statistics available for solar cycle 23, the calculated success rate at high solar wind speed, although clearly above 50 %, was indicative rather than conclusive. The RMS error estimated for shock arrival times for every cycle phase and for the composite sample was in each case significantly better than would be expected for a random data set. Also, the parameter "Probability of Detection, yes" (PODy) which presents the Proportion of Yes observations that were correctly forecast (i.e. the ratio between the shocks correctly predicted and all the shocks observed), yielded values for the rise/maximum/decay phases of the cycle and using the composite sample of 0.85, 0.64, 0.79 and 0.77, respectively. The statistical results obtained through detailed analysis of the available data provided insights into how changing circumstances on the Sun and in interplanetary space can affect the performance of the model. Since shock arrival predictions are widely utilized in making commercially significant decisions re. protecting space assets, the present detailed archival studies can be useful in future operational decision making during solar cycle 24. It would be of added value in this context to use Briggs-Rupert methodology to estimate the cost to an operator of acting on an incorrect forecast.

Shock Propagation Model version 2 and its application in predicting the arrivals at Earth of interplanetary shocks during Solar Cycle 23

“…The 2×2 “contingency table” is often used in the evaluation of meteorological models [e.g., Schaefer , ]. It provides information about the success or failure of the forecast in the data set and has been employed in Fearless Forecast models [ Fry et al , ; McKenna‐Lawlor et al , , ; Smith et al , ]. In this paper, we follow the fundamental definitions adopted in the Fearless Forecasts, namely, if a shock is predicted to arrive and actually observed within ± 24 h, then the prediction is called a “Hit” (h); if a shock is detected, but predicted not to arrive, or predicted to arrive 24 h away from the detection time, this prediction is called a “Miss” (m); if a shock is predicted to arrive, but no shock is observed 1–5 days after the solar event, then the prediction is called a “False alarm” (fa); and if a shock is neither predicted to arrive nor detected 1–5 days after the solar event, then this prediction is called a “Correct null” (cn).…”

“…These skill scores include (1) probability of detection, yes, PODy = a/(a + c); (2) probability of detection, no, PODn = d/(b + d); (3) false alarm ratio, FAR = b/(a + b); (4) bias, BIAS = (a + b)/(a + c); (5) critical success index, CSI = a/(a + b + c); (6) true skill statistic, TSS = PODy + PODn − 1; (7) Heidke skill score, HSS = (a + d − C1)/(N − C1), here C1 = C2 + (b + d)(c + d)/N, C2 = (a + c)(a + b)/N; (8) Gilbert skill score, GSS = (a − C2)/(a + b + c − C2); (9) success rate, SR = (a + d)/N. Details about the definitions and applications of these skill scores can be found in Schaefer [], Mozer and Briggs [], Smith et al [], and McKenna‐Lawlor et al []. The specific parameter of interest to a user depends on his/her purpose.…”

“…However, it should be pointed out that among these various standard parameters, only PODn, TSS, HSS, and SR involve correct nulls. These forecast skill scores have earlier been used to evaluate the Fearless Forecast models [e.g., Fry et al , ; McKenna‐Lawlor et al , , ; Smith et al , ]. Table gives a statistical comparison of these skill scores derived by SPM2 with those derived by HAFv.2 based on 551 rather similar events, with those by STOA on 463 events, and with those by ISPM on 418 events.…”

“…The prediction success rate of SPM2 is slightly higher (SR = 0.61) than that of HAFv.2 (SR = 0.54) for the total 551 events. It should be noted that McKenna‐Lawlor et al [] showed for a sample of 584 events that the parameter PODy yielded values for the rise/maximum/decay phases of Solar Cycle 23 and when using the composite sample of 0.85, 0.64, 0.79, and 0.77. Also, that the parameters SR obtained were correspondingly only 0.53, 0.51, 0.59, and 0.55.…”

“…Both empirical and physics‐based models have been proposed to predict shock arrival times (SATs) based on the relationship between features of solar disturbances and their corresponding IP shocks. By using available solar data as input parameters, these physics‐based models have been employed to give “near real‐time” predictions of shock arrivals at Earth, such as the “Shock Time of Arrival” (STOA) Model [ Dryer and Smart , ; Smart and Shea , , ], the “Interplanetary Shock Propagation” Model (ISPM) [ Smith and Dryer , , ], and the “Hakamada‐Akasofu‐Fry version 2” (HAFv.2) model [ Dryer et al , , ; Fry et al , , , ; McKenna‐Lawlor et al , , , ; Smith et al , , ; Sun et al , , , ]. The STOA, ISPM, and HAFv.2 models use similar input solar parameters, including the source location of the associated flare, the start time of the metric Type II radio burst, the proxy piston‐driving time duration, and the background solar wind speed.…”