Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure

Traditional machine learning (ML) metrics overestimate model performance for materials discovery. We introduce (1) leave-onecluster-out cross-validation (LOCO CV) and (2) a simple nearestneighbor benchmark to show that model performance in discovery applications strongly depends on the problem, data sampling, and extrapolation. Our results suggest that ML-guided iterative experimentation may outperform standard high-throughput screening for discovering breakthrough materials like high-T c superconductors with ML.Materials informatics (MI), or the application of data-driven algorithms to materials problems, has grown quickly as a field in recent years. 9 Across all of these applications, a training database of simulated or experimentally-measured materials properties serves as input to a ML algorithm that predictively maps features (i.e., materials descriptors) to target materials properties. Ideally, the result of training such models would be the experimental realization of new materials with promising properties. The MI community has produced several such success stories, including thermoelectric compounds, 10,11 shapememory alloys, 12 superalloys, 13 and 3d-printable high-strength aluminum alloys. 14 However, in many cases, a model is itself the output of a study, and the question becomes: to what extent could the model be used to drive materials discovery? Typically, the performance of ML models of materials properties is quantified via cross-validation (CV). CV can be performed either in a single division of the available data into a training set (to build the model) and a test set (to evaluate its performance), or as an ensemble process known as k-fold CV wherein the data are partitioned into k nonoverlapping subsets of nearly equal size (folds) and model performance is averaged across each combination of k-1 training folds and one test fold. Leave-one-out crossvalidation (LOOCV) is the limit where k is the number of total examples in the dataset. Table 1 summarizes some examples of model performance statistics as reported in the aforementioned studies (some studies involved testing multiple algorithms across multiple properties).In Table 1, the reported model performance is uniformly excellent across all studies. A tempting conclusion is that any of these models could be used for one-shot high-throughput screening of large numbers of materials for desired properties. However, as we discuss below, traditional CV has critical shortcomings in terms of quantifying ML model performance for materials discovery. Issues with traditional crossvalidation for materials discoveryMany ML benchmark problems consist of data classification into discrete bins, i.e., pattern matching. For example, the Design, System, ApplicationMachine learning (ML) has become a widely-adopted predictive tool for materials design and discovery. Random k-fold cross-validation (CV), the traditional gold-standard approach for evaluating the quality of ML models, is fundamentally mismatched to the nature of materials discovery, and leads to ...

show abstract

“…The above considerations are well understood in other domains, such as ecology, 21 where a variety of schemes exist to…”

Section: Materials Informatics (Mi)mentioning

confidence: 99%

Can machine learning identify the next high-temperature superconductor? Examining extrapolation performance for materials discovery

Meredig

Antono

Church

et al. 2018

Mol. Syst. Des. Eng.

204

180

View full text Add to dashboard Cite

show abstract

“…Spatial disaggregation has been shown to be a reasonable method to avoid the excessive overconfidence that can possibly result from other training and testing methodologies of spatial models (Bahn and McGill, 2013;Roberts et al, 2017). The cross-validation method was as follows.…”

Section: Environmental Predictors For Random Forest Models and Modelmentioning

confidence: 99%

“…The cross-validation method was as follows. We chose to use the spatial blocking method from Roberts et al (2017). This places data into consistently sized and spatially separate blocks or bins.…”

Section: Environmental Predictors For Random Forest Models and Modelmentioning

confidence: 99%

A synthetic map of the north-west European Shelf sedimentary environment for applications in marine science

Wilson

Speirs

Sabatino

et al. 2018

Earth Syst. Sci. Data

View full text Add to dashboard Cite

Abstract. Seabed sediment mapping is important for a wide range of marine policy, planning and scientific issues, and there has been considerable national and international investment around the world in the collation and synthesis of sediment datasets. However, in Europe at least, much of this effort has been directed towards seabed classification and mapping of discrete habitats. Scientific users often have to resort to reverse engineering these classifications to recover continuous variables, such as mud content and median grain size, that are required for many ecological and biophysical studies. Here we present a new set of 0.125 • by 0.125 • resolution synthetic maps of continuous properties of the north-west European sedimentary environment, extending from the Bay of Biscay to the northern limits of the North Sea and the Faroe Islands. The maps are a blend of gridded survey data, statistically modelled values based on distributions of bed shear stress due to tidal currents and waves, and bathymetric properties. Recent work has shown that statistical models can predict sediment composition in British waters and the North Sea with high accuracy, and here we extend this to the entire shelf and to the mapping of other key seabed parameters. The maps include percentage compositions of mud, sand and gravel; porosity and permeability; median grain size of the whole sediment and of the sand and the gravel fractions; carbon and nitrogen content of sediments; percentage of seabed area covered by rock; mean and maximum depth-averaged tidal velocity and wave orbital velocity at the seabed; and mean monthly natural disturbance rates. A number of applications for these maps exist, including species distribution modelling and the more accurate representation of sea-floor biogeochemistry in ecosystem models. The data products are available from https://doi.org/10.15129/1e27b806-1eae-494d-83b5-a5f4792c46fc.

show abstract

“…Due to the nested structure of our dataset and considering the 12 plots as independent, the GoFs were obtained by applying a leave-one-plot-out cross-validation (LOPOCV) procedure [33]. During LOPOCV, we did not repeat the metrics selection procedure, such that our cross-validation approach did not account for any potential errors caused by model mis-specification.…”

Section: Model Validationmentioning

confidence: 99%

Predicting Selected Forest Stand Characteristics with Multispectral ALS Data

et al. 2018

View full text Add to dashboard Cite

Abstract:In this study, the potential of multispectral airborne laser scanner (ALS) data to model and predict some forest characteristics was explored. Four complementary characteristics were considered, namely, aboveground biomass per hectare, Gini coefficient of the diameters at breast height, Shannon diversity index of the tree species, and the number of trees per hectare. Multispectral ALS data were acquired with an Optech Titan sensor, which consists of three scanners, called channels, working in three wavelengths (532 nm, 1064 nm, and 1550 nm). Standard ALS data acquired with a Leica ALS70 system were used as a reference. The study area is located in Southern Norway, in a forest composed of Scots pine, Norway spruce, and broadleaf species. ALS metrics were extracted for each plot from both elevation and intensity values of the ALS points acquired with both sensors, and for all three channels of the ALS multispectral sensor. Regression models were constructed using different combinations of metrics. The results showed that all four characteristics can be accurately predicted with both sensors (the best R 2 being greater than 0.8), but the models based on the multispectral ALS data provide more accurate results. There were differences regarding the contribution of the three channels of the multispectral ALS. The models based on the data of the 532 nm channel seemed to be the least accurate.

show abstract

Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure

Cited by 1,362 publications

References 97 publications

Can machine learning identify the next high-temperature superconductor? Examining extrapolation performance for materials discovery

Can machine learning identify the next high-temperature superconductor? Examining extrapolation performance for materials discovery

A synthetic map of the north-west European Shelf sedimentary environment for applications in marine science

Predicting Selected Forest Stand Characteristics with Multispectral ALS Data

Contact Info

Product

Resources

About