Performance assessment of NOSTRADAMUS & other machine learning-based telemetry monitoring systems on a spacecraft anomalies database

Abstract

Spacecraft health monitoring on ground is commonly performed using two complementary methods: a short-term automatic Out-Of-Limits (OOL) verification after each new telemetry reception, and a long-term monitoring using statistical features (e.g. daily minimum, mean and maximum). In the past few years, various new monitoring methods based on machine learning have been suggested in literature, with a great interest for their new detection capabilities, already demonstrated for some use-cases or even for operational use. These methods differ not only by their mathematical core, but also by their data preprocessing and by the tuning of their control parameters, raising some issues for quantitative performance comparison. Indeed, benchmarks of machine-learning algorithms with classical datasets can be found in the literature, but it is no longer the case once they are embedded in a complete spacecraft monitoring system. This paper presents a methodology for such a comparison on an anomalies database constituted with housekeeping telemetry of CNES' operated satellites. The performance of NOSTRADAMUS system used at CNES, which principle and operational use are also described in this paper, is compared to the one of algorithms inspired from literature such as Novelty Detection (ESOC), Project Sybil, and ATHMoS (DLR).

A critical review on the state-of-the-art and future prospects of machine learning for Earth observation operations

“…Ranasinghe et al provides a comprehensive analysis of FDIR (Ranasinghe et al, 2022). Fuertes et al (Fuertes et al, 2018) discuss ML-based fault detection using NOSTRADAMUS, an algorithm developed by the Centre National des Études Spatiales (CNES). NOSTRADAMUS uses a One-Class -Support Vector Machine (OC-SVM), a common algorithm used to detect outliers, to detect the presence of an anomaly in telemetry data.…”

A Survey on Anomaly Detection and Diagnosis Problem in the Space System Operation

“…Fuertes, Sylvain, Barbara Pilastre, and Stéphane D'Escrivan (Fuertes et al, 2018) have compared three unsupervised algorithms: One-Class Support Vector Machine (OC-SVM is a type of Support Vector Machines) (Rana, Divya, 2015), Density-based spatial clustering of applications with noise (DBSCAN) and k-Nearest Neighbors algorithm. They have obtained approximate results.…”

“…results are compared to each other. (Fuertes et al, 2018) Clustering (Unsupervised Learning) One-Class Support Vector Machine (OC-SVM), k-Nearest Neighbors algorithm, DBSCAN CNES operated satellite data (+) High accuracy, but depends on the application context and feature selections.…”