Validating Intensity Prediction Equations for Italy by Observations

Abstract

The engineering seismology community has recently recognized the importance of validating the performance of predictive models for seismic hazard by independent observations, yielding a number of studies on the relative performance of ground-motion prediction equations. The validation of intensity prediction equations (IPEs) has attracted less attention. We fill this gap by validating eight Italian IPEs plus one global IPE using five sets of Italian macroseismic intensity data, of which three are prospective and two retrospective to the models. We implemented multiple scoring methods to validate the models and found that the simple score of mean absolute error is sufficient to measure the general model performance. Good models consistently perform well under multiple methods and datasets, showing robustness. Models with physical functional forms are found to perform better. The global IPE performed well for Italian data, implying insignificant regional differences for IPEs. This result encourages grouping intensity data collected from multiple geographic regions, both from the Internet and traditional surveys, into a larger dataset for the use of future model development and validation.

Measuring the Performance of Ground‐Motion Models: The Importance of Being Independent

“…"foreign models" indicates that foreign models are evaluated using local tests, so the data are naturally independent of the models; similarly for "stochastic models." † Regarding Mak et al (2015), the dataset for small and moderate (respectively, large) earthquake data was independent (respectively, not independent) of the evaluated models.…”

The Predictive Power of Ground‐Motion Prediction Equations

“…This aspect is of particular relevance for GMPEs, which are generally based on complex functional forms with several parameters. As observed by Scherbaum et al (2009), despite the strong overfitting to which the models are exposed, their performance in predicting new data is seldom addressed (e.g., Kuehn et al, 2009;Kaklamanos and Baise, 2011;Mak et al, 2015), and rarely during the calibration phase. Indeed, a GMPE performance is generally evaluated through residual analysis performed with respect to the same calibration data, and the significance of the model coefficients as well as their trade-offs are often not discussed.…”

Reply to “Comment on ‘Validating Intensity Prediction Equations for Italy by Observations’ by Sum Mak, Robert Alan Clements, and Danijel Schorlemmer” by Mathias Raschke

“…Recently, Roselli et al (2016) also made a similar proposal of using the BIC on ground-motion prediction equation (GMPE) evaluation, although they frankly admitted that GMPE evaluations were seldom conducted on the same set of data used to calibrate the models, and so the strict mathematical meaning for likelihood-based selection criteria (including Akaike information criterion, BIC, and others) to penalize excessive degrees of freedom was not achieved. This situation also applies to our IPE evaluation, even for the retrospective datasets we used (DBMI11 RM and DBMI11 RL ; see table 4 of Mak et al, 2015): The retrospective datasets were (at least partially) included in the data used by the modelers to develop their IPEs, but this did not mean that the IPEs were developed using exactly the retrospective datasets. In fact, by only the information published by the modelers, it is usually impossible to reconstruct exactly the dataset the modelers used to calibrate their models.…”

“…A major reason for us to use the nonparametric score of weighted mean absolute error (wMAE), as explained in the Appropriateness of the Score section of Mak et al (2015), and regarded by Raschke as "not a plausible reason," was that not all IPEs we evaluated have provided the necessary parameters to implement a probabilistic score. Models 1987GPT, 1996P, and 2000P (see Mak et al, 2015) did not provide the information on the uncertainty σ of their predictions. When we implemented the LLH ( fig.…”

“…When we implemented the LLH ( fig. S13 of Mak et al, 2015), we had to assign a σ to the three models. We therefore mentioned the implementation was tentative.…”