A cautionary note regarding comparisons of fire danger indices

Abstract

Abstract. Over the past decade, several methods have been used to compare the performance of fire danger indices in an effort to find the most appropriate indices for particular regions or circumstances. Various authors have proposed comparators and demonstrated different responses of indices to their tests, but rarely has much effort been put into demonstrating the validity of the comparators themselves. We present a demonstration that many of the published comparators are sensitive to the different frequency distributions, that may be inherent in the performance of the different indices, and outline a non-parametric method that may be useful for future work. We compare four hypothetical fire danger indices, three of which are simple mathematical transformations of each other. The hypothesis tested is that the comparators often used in such studies may indicate spurious performance differences between these indices, which is found to be the case. Non-parametric methods are robust to differences in index value frequency distribution and may allow more valid comparisons of fire danger indices. The new comparison method is shown to have advantages over other non-parametric comparators.

Calibration and evaluation of the Canadian Forest Fire Weather Index (FWI) System for improved wildland fire danger rating in the United Kingdom

“…These data were explored in detail for a number of particularly "extreme" wildfire incidents (Sect. 4.3.1), and then an intercomparison of the components was performed using the ranked percentile curve approach of Eastaugh et al (2012) to identify the components that best highlight fire danger in the UK (Sect. 4.3.2).…”

“…A percentile-based evaluation method was appealing for such a comparison, since these data were readily available to us and they are uninfluenced by the differences in frequency distributions and scales of the raw components. Comparing differences in percentiles on fire/nonfire days between indices, as used previously by Andrews et al (2003), can form a simple yet effective evaluation method, but the choice of percentiles for evaluation can influence which index is considered to have greatest skill (Eastaugh et al, 2012). Therefore, we elected to use the "ranked percentile curve" approach devised in the Eastaugh et al (2012) review of fire danger index comparators.…”

“…Comparing differences in percentiles on fire/nonfire days between indices, as used previously by Andrews et al (2003), can form a simple yet effective evaluation method, but the choice of percentiles for evaluation can influence which index is considered to have greatest skill (Eastaugh et al, 2012). Therefore, we elected to use the "ranked percentile curve" approach devised in the Eastaugh et al (2012) review of fire danger index comparators. This method has subsequently been applied by Arpaci et al (2013) and Eastaugh and Hasenauer (2014).…”

“…This method has subsequently been applied by Arpaci et al (2013) and Eastaugh and Hasenauer (2014). A brief description of the ranked percentile curve approach of Eastaugh et al (2012) is provided here. For fire danger indices with a daily time resolution, all index values are first converted to percentiles, and the percentiles on days on which fires occurred ("fire days") are extracted and plotted by ascending rank to create a ranked percentile curve.…”

“…Such episodes place extreme stress upon the resources of fire and rescue services (FRS), in terms of both personnel and firefighting response assets . Accordingly, use of a fire danger rating system (FDRS) to forecast when and where these wildfire episodes are most likely is of growing interest to FRS and government agencies, as a sufficiently reliable operational system could enhance short-term wildfire response planning and resource allocation (Eastaugh et al, 2012).…”