Receiver operating characteristics (ROC) as validation tool for prospectivity models — A magmatic Ni–Cu case study from the Central Lapland Greenstone Belt, Northern Finland

Nykänen, Vesa; Lahti, I.; Niiranen, Tero; Korhonen, Kimmo

doi:10.1016/j.oregeorev.2014.09.007

Cited by 152 publications

(33 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, AUC has also been established as a recommended metric for PU-learning problems (Elkan and Noto 2008;Jain et al 2017). Thus, AUC is a natural performance measure for MPM classifier evaluation, and studies such as Brown et al (2003), Nykänen (2008), Nykänen et al (2015) and Rodriguez-Galiano et al (2015) have used AUC for evaluating MPM models. Further, since adequately large separate test data may not be available for MPM, cross-validation (CV) is necessary for validating the models (see e.g., Abedi et al 2012;Rigol-Sanchez et al 2003;Carranza 2008;Rodriguez-Galiano et al 2015).…”

Section: Introductionmentioning

confidence: 99%

The spatial leave-pair-out cross-validation method for reliable AUC estimation of spatial classifiers

Airola

Pohjankukka

Torppa

et al. 2018

Data Min Knowl Disc

Self Cite

View full text Add to dashboard Cite

Machine learning based classification methods are widely used in geoscience applications, including mineral prospectivity mapping. Typical characteristics of the data, such as small number of positive instances, imbalanced class distributions and lack of verified negative instances make ROC analysis and cross-validation natural choices for classifier evaluation. However, recent literature has identified two sources of bias, that can affect reliability of area under ROC curve estimation via cross-validation on spatial data. The pooling procedure performed by methods such as leave-one-out can introduce a substantial negative bias to results. At the same time, spatial dependencies leading to spatial autocorrelation can result in overoptimistic results, if not corrected for. In this work, we introduce the spatial leave-pair-out cross-validation method, that corrects for both of these biases simultaneously. The methodology is used to benchmark a number of classification methods on mineral prospectivity mapping data from the Central Lapland greenstone belt. The evaluation highlights the dangers of obtaining misleading results on spatial data and demonstrates how these problems can be avoided. Further, the results show the advantages of simple linear models for this classification task.

show abstract

Section: Introductionmentioning

confidence: 99%

The spatial leave-pair-out cross-validation method for reliable AUC estimation of spatial classifiers

Airola

Pohjankukka

Torppa

et al. 2018

Data Min Knowl Disc

Self Cite

View full text Add to dashboard Cite

show abstract

“…The subjective nature of fuzzy set theory and the Fuzzy Logic method can be circumvented by refining input variables using the ROC curve tool developed by Nykänen et al (2015Nykänen et al ( , 2017. The approach provides a quantitative metric for assessing subjective aspects of the Fuzzy Logic technique, namely the application of the fuzzy membership function and fuzzy operators such as FuzzyOR (An et al, 1991;Bonham-Carter, 1994).…”

Section: Fuzzy Membership Transformationmentioning

confidence: 99%

“…Various MLAs are available for prospectivity modelling, however, it is the Random Forest algorithm that has consistently proven to be highly effective in comparison to Support Vector Machines and Artificial Neural Networks Laborte, 2015a, 2015b;Rodriguez-Galiano et al, 2015;Carranza and Laborte, 2016;Sun et al, 2019). For this reason, two Random Forest models are presented for prospectivity modelling, using: (i) standardized input variables with no transformation; (ii) variables transformed using the guided fuzzy set theory approach of Nykänen et al (2015Nykänen et al ( , 2017).…”

Section: Machine Learning For Prospectivity Modellingmentioning

confidence: 99%

“…The advantage of this is the ability to weight different data and to introduce some dependencies between variables that may be inferred by the user but not captured in the data within a conceptual deposit model. Until recently, this technique has been considered highly subjective, but work by Nykänen et al (2015Nykänen et al ( , 2017 provides a means of guiding the data processing by iteratively tuning evidence layers using an evaluation metric. Another method by Burkin et al (2019) incorporates feature evidence into the initial evidence layer to mitigate interpretative bias of the conceptual model by the user.…”

Section: Introductionmentioning

confidence: 99%

“…In this study we use fuzzy transformations as part of the feature extraction step in MLA modelling. We take the approach of Nykänen et al (2015Nykänen et al ( , 2017 to ensure the user-knowledge that is introduced to potentially improve a data-driven analysis is quantifiable.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A machine learning approach to tungsten prospectivity modelling using knowledge-driven feature extraction and model confidence

Yeomans¹,

Shail²,

Grebby³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Novel mineral prospectivity modelling presented here applies knowledge-driven feature extraction to a data-driven machine learning approach for tungsten mineralisation. The method emphasises the importance of appropriate model evaluation and develops a new Confidence Metric to generate spatially refined and robust exploration targets. The data-driven Random Forest™ algorithm is employed to model tungsten mineralisation in SW England using a range of geological, geochemical and geophysical evidence layers which include a depth to granite evidence layer. Two models are presented, one using standardised input variables and a second that implements fuzzy set theory as part of an augmented feature extraction step. The use of fuzzy data transformations mean feature extraction can incorporate some user-knowledge about the mineralisation into the model. The typically subjective approach is guided using the Receiver Operating Characteristics (ROC) curve tool where transformed data are compared to known training samples. The modelling is conducted using 34 known true positive samples with 10 sets of randomly generated true negative samples to test the random effect on the model. The two models have similar accuracy but show different spatial distributions when identifying highly prospective targets. Areal analysis shows that the fuzzy-transformed model is a better discriminator and highlights three areas of high prospectivity that were not previously known. The Confidence Metric, derived from model variance, is employed to further evaluate the models. The new metric is useful for refining exploration targets and highlighting the most robust areas for follow-up investigation. The fuzzy-transformed model is shown to contain larger areas of high model confidence compared to the model using standardised variables. Finally, legacy mining data, from drilling reports and mine descriptions, is used to further validate the fuzzy-transformed model and gauge the depth of potential deposits. Descriptions of mineralisation corroborate that the targets generated in these models could be undercover at depths of less than 300 m. In summary, the modelling workflow presented herein provides a novel integration of knowledge-driven feature extraction with data-driven machine learning modelling, while the newly derived Confidence Metric generates reliable mineral exploration targets.

show abstract

Spatial Land Use Change Modeling Techniques

Al-sharif

Pradhan

Abdullahi

2017

Spatial Modeling and Assessment of Urban Form

View full text Add to dashboard Cite

Receiver operating characteristics (ROC) as validation tool for prospectivity models — A magmatic Ni–Cu case study from the Central Lapland Greenstone Belt, Northern Finland

Cited by 152 publications

References 47 publications

The spatial leave-pair-out cross-validation method for reliable AUC estimation of spatial classifiers

The spatial leave-pair-out cross-validation method for reliable AUC estimation of spatial classifiers

A machine learning approach to tungsten prospectivity modelling using knowledge-driven feature extraction and model confidence

Spatial Land Use Change Modeling Techniques

Contact Info

Product

Resources

About