RMSE is not enough: Guidelines to robust data-model comparisons for magnetospheric physics

Liemohn, Michael W.; Shane, A. D.; Azari, Abigail; Petersen, A. K.; Swiger, B. M.; Mukhopadhyay, Agnit

doi:10.1016/j.jastp.2021.105624

Cited by 68 publications

(50 citation statements)

References 122 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Because modeled MPSD and CPCP were compared against multiple datasets, the lone usage of error metrics like RMSE is not enough to meaningfully rank model performance (Liemohn et al, 2021) as has often been done before (e.g., Pulkkinen et al, 2011). Because there is no single right answer, a significant aim of this study has been to develop innovative metrics to better quantify the performance of global models against multiple, divergent observationally-derived estimates.…”

Section: Discussionmentioning

confidence: 99%

Global Magnetohydrodynamic Simulations: Performance Quantification of Magnetopause Distances and Convection Potential Predictions

Mukhopadhyay

Jia

Welling

et al. 2021

Front. Astron. Space Sci.

Self Cite

View full text Add to dashboard Cite

The performance of three global magnetohydrodynamic (MHD) models in estimating the Earth's magnetopause location and ionospheric cross polar cap potential (CPCP) have been presented. Using the Community Coordinated Modeling Center's Run-on-Request system and extensive database on results of various magnetospheric scenarios simulated for a variety of solar weather patterns, the aforementioned model predictions have been compared with magnetopause standoff distance estimations obtained from six empirical models, and with cross polar cap potential estimations obtained from the Assimilative Mapping of Ionospheric Electrodynamics (AMIE) Model and the Super Dual Auroral Radar Network (SuperDARN) observations. We have considered a range of events spanning different space weather activity to analyze the performance of these models. Using a fit performance metric analysis for each event, the models' reproducibility of magnetopause standoff distances and CPCP against empirically-predicted observations were quantified, and salient features that govern the performance characteristics of the modeled magnetospheric and ionospheric quantities were identified. Results indicate mixed outcomes for different models during different events, with almost all models underperforming during the extreme-most events. The quantification also indicates a tendency to underpredict magnetopause distances in the absence of an inner magnetospheric model, and an inclination toward over predicting CPCP values under general conditions.

show abstract

Section: Discussionmentioning

confidence: 99%

Global Magnetohydrodynamic Simulations: Performance Quantification of Magnetopause Distances and Convection Potential Predictions

Mukhopadhyay

Jia

Welling

et al. 2021

Front. Astron. Space Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Based on a rule of thumb, RMSE values less than 0.1 shows that the model can relatively predict the data accurately. Furthermore, a RMSE value of the present study (0.016) that is smaller than both standard deviation values of observed data (0.305) and predicted data (0.313) is considered good [27].…”

Section: Resultsmentioning

confidence: 54%

Validating a low-cost seismometer using a shaking table

et al. 2022

View full text Add to dashboard Cite

Ambient vibration measurement using a seismometer represent a non-invasive and non-destructive method for seismic-related studies. This method inhibits widespread applications, i.e., assessing site fundamental frequency, estimating bedrock level, calculating building natural frequency, deducing shear wave velocity profile. However, the instrument required to carry out such ambient vibration measurement is generally expensive. Recently, there is a low-cost Raspberry Shake 3D seismometer (RS-3D) available on the market. This seismometer is based on Raspberry Pi hardware. It contains three 4.5 Hz orthogonally oriented geophones, which can be set at a sampling rate of 100 Hz and a bandwidth of -3dB points at 0.6 to 34 Hz. Thus, laboratory testing was carried out to assess this device’s performance in this demanding application. Generally, RS-3D demonstrates a great performance. Further detailed results are presented in this paper.

show abstract

“…This coefficient is between 1 and −1 and if there is no relationship between the two variables, it is equal to zero [60]. These indicators are calculated based on the following formulas [61][62][63][64]:…”

Section: Validation Indicatorsmentioning

confidence: 99%

Hybridization of Differential Evolution and Adaptive-NetworkBased Fuzzy Inference System in Estimation of Compression Coefficient of Plastic Clay Soil

Nguyen¹,

Hai²,

Al‐Ansari³

et al. 2022

Computer Modeling in Engineering &Amp; Sciences

View full text Add to dashboard Cite

One of the important geotechnical parameters required for designing of the civil engineering structure is the compressibility of the soil. In this study, the main purpose is to develop a novel hybrid Machine Learning (ML) model (ANFIS-DE), which used Differential Evolution (DE) algorithm to optimize the predictive capability of Adaptive-Network-based Fuzzy Inference System (ANFIS), for estimating soil Compression coefficient (Cc) from other geotechnical parameters namely Water Content, Void Ratio, Specific Gravity, Liquid Limit, Plastic Limit, Clay content and Depth of Soil Samples. Validation of the predictive capability of the novel model was carried out using statistical indices: Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Correlation Coefficient (R). In addition, two popular ML models namely Reduced Error Pruning Trees (REPTree) and Decision Stump (Dstump) were used for comparison. Results showed that the performance of the novel model ANFIS-DE is the best (R = 0.825, MAE = 0.064 and RMSE = 0.094) in comparison to other models such as REPTree (R = 0.7802, MAE = 0.068 and RMSE = 0.0988) and Dstump (R = 0.7325, MAE = 0.0785 and RMSE = 0.1036). Therefore, the ANFIS-DE model can be used as a promising tool for the correct and quick estimation of the soil Cc, which can be employed in the design and construction of civil engineering structures.

show abstract

RMSE is not enough: Guidelines to robust data-model comparisons for magnetospheric physics

Cited by 68 publications

References 122 publications

Global Magnetohydrodynamic Simulations: Performance Quantification of Magnetopause Distances and Convection Potential Predictions

Global Magnetohydrodynamic Simulations: Performance Quantification of Magnetopause Distances and Convection Potential Predictions

Validating a low-cost seismometer using a shaking table

Hybridization of Differential Evolution and Adaptive-NetworkBased Fuzzy Inference System in Estimation of Compression Coefficient of Plastic Clay Soil

Contact Info

Product

Resources

About