Uncertainty quantification techniques for data-driven space weather modeling: thermospheric density application

Abstract

Machine learning (ML) has been applied to space weather problems with increasing frequency in recent years, driven by an influx of in-situ measurements and a desire to improve modeling and forecasting capabilities throughout the field. Space weather originates from solar perturbations and is comprised of the resulting complex variations they cause within the numerous systems between the Sun and Earth. These systems are often tightly coupled and not well understood. This creates a need for skillful models with knowledge about the confidence of their predictions. One example of such a dynamical system highly impacted by space weather is the thermosphere, the neutral region of Earth’s upper atmosphere. Our inability to forecast it has severe repercussions in the context of satellite drag and computation of probability of collision between two space objects in low Earth orbit (LEO) for decision making in space operations. Even with (assumed) perfect forecast of model drivers, our incomplete knowledge of the system results in often inaccurate thermospheric neutral mass density predictions. Continuing efforts are being made to improve model accuracy, but density models rarely provide estimates of confidence in predictions. In this work, we propose two techniques to develop nonlinear ML regression models to predict thermospheric density while providing robust and reliable uncertainty estimates: Monte Carlo (MC) dropout and direct prediction of the probability distribution, both using the negative logarithm of predictive density (NLPD) loss function. We show the performance capabilities for models trained on both local and global datasets. We show that the NLPD loss provides similar results for both techniques but the direct probability distribution prediction method has a much lower computational cost. For the global model regressed on the Space Environment Technologies High Accuracy Satellite Drag Model (HASDM) density database, we achieve errors of approximately 11% on independent test data with well-calibrated uncertainty estimates. Using an in-situ CHAllenging Minisatellite Payload (CHAMP) density dataset, models developed using both techniques provide test error on the order of 13%. The CHAMP models—on validation and test data—are within 2% of perfect calibration for the twenty prediction intervals tested. We show that this model can also be used to obtain global density predictions with uncertainties at a given epoch.

Science Through Machine Learning: Quantification of Post‐Storm Thermospheric Cooling

“…The 10 best models are evaluated on the training and validation sets to determine the final model considering the lowest errors. The model development for CHAMP, HASDM, and subsequently JB2008 is outlined by Licata and Mehta (2022) with the only difference being the SYM‐H time series inputs for CHAMP. We provide additional details on model development in Appendices and .…”

“…This study used existing models from Licata et al. (2022) and Licata and Mehta (2022) which were developed with the goal of forecasting in mind. This led to the decision to include the geomagnetic index time series as ML model inputs.…”

Science through Machine Learning: Quantification of Poststorm Thermospheric Cooling

“…The ten best models are evaluated on the training and validation sets to determine the final model considering the lowest errors. The model development for CHAMP, HASDM, and subsequently JB2008 is outlined by Licata and Mehta [40] with the only difference being the SYM-H time series inputs for CHAMP. We provide additional details on model development in Appendices A and B.…”

“…For this study, a mean square error (MSE) loss function would suffice since the goal is to develop a model with minimal error with respect to the respective dataset. However, we used models from previous work, Licata et al [40], which used the negative logarithm of predictive density (NLPD) loss function,…”

Machine Learning in Orbit Estimation: a Survey

“…With this data it was possible to train a NN to replicate the HASDM model [107] using the same input, so as to add uncertainty information through the use of Monte-Carlo Dropout. Further advancements were accomplished by the use of Bayesian Neural Networks on the global HASDM dataset and the local CHAMP dataset [108]. Bayesian Neural Networks have recently garnered more attention in the machine learning world, as they allow for a more principled understanding of uncertainty, when compared to Monte-Carlo Dropout or Ensembles.…”