Conformal Predictions for Hybrid System State Classification

Bortolussi, Luca; Cairoli, Francesca; Paoletti, Nicola; Stoller, Scott D.

doi:10.1007/978-3-030-31514-6_13

Cited by 3 publications

(3 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [8], we presented a preliminary version of this approach. The present paper greatly extends and improves that work by including an automated and optimal method to select rejection thresholds, the active learning method, and an evaluation on larger HA benchmarks.…”

Section: Related Workmentioning

confidence: 99%

Neural Predictive Monitoring

Bortolussi

Cairoli

Paoletti

et al. 2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Neural State Classification (NSC) is a recently proposed method for runtime predictive monitoring of Hybrid Automata (HA) using deep neural networks (DNNs). NSC trains a DNN as an approximate reachability predictor that labels a given HA state x as positive if an unsafe state is reachable from x within a given time bound, and labels x as negative otherwise. NSC predictors have very high accuracy, yet are prone to prediction errors that can negatively impact reliability. To overcome this limitation, we present Neural Predictive Monitoring (NPM), a technique based on NSC and conformal prediction that complements NSC predictions with statistically sound estimates of uncertainty. This yields principled criteria for the rejection of predictions likely to be incorrect, without knowing the true reachability values. We also present an active learning method that significantly reduces both the NSC predictor's error rate and the percentage of rejected predictions. Our approach is highly efficient, with computation times on the order of milliseconds, and effective, managing in our experimental evaluation to successfully reject almost all incorrect predictions.

show abstract

Section: Related Workmentioning

confidence: 99%

Neural Predictive Monitoring

Bortolussi

Cairoli

Paoletti

et al. 2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, in a binary classification problem, one can consider the difference between the probability of the two classes as a measure of uncertainty, where small differences indicate uncertain predictions. Previous experiments in [29] have shown that such a measure yields poor error detection in NPM, because the measure is overconfident in predictions that turn out being erroneous.…”

Section: Remark 3 (Softmax Probabilities As Uncertainty Measures)mentioning

confidence: 99%

“…In [29], we presented a preliminary version of the frequentist approach. In [10], we added to it an automated and optimal method to select the rejection thresholds and an active learning framework.…”

Section: Related Workmentioning

confidence: 99%

Neural predictive monitoring and a comparison of frequentist and Bayesian approaches

Bortolussi

Cairoli

Paoletti

et al. 2021

Int J Softw Tools Technol Transfer

Self Cite

View full text Add to dashboard Cite

Neural state classification (NSC) is a recently proposed method for runtime predictive monitoring of hybrid automata (HA) using deep neural networks (DNNs). NSC trains a DNN as an approximate reachability predictor that labels an HA state x as positive if an unsafe state is reachable from x within a given time bound, and labels x as negative otherwise. NSC predictors have very high accuracy, yet are prone to prediction errors that can negatively impact reliability. To overcome this limitation, we present neural predictive monitoring (NPM), a technique that complements NSC predictions with estimates of the predictive uncertainty. These measures yield principled criteria for the rejection of predictions likely to be incorrect, without knowing the true reachability values. We also present an active learning method that significantly reduces the NSC predictor’s error rate and the percentage of rejected predictions. We develop two versions of NPM based, respectively, on the use of frequentist and Bayesian techniques to learn the predictor and the rejection rule. Both versions are highly efficient, with computation times on the order of milliseconds, and effective, managing in our experimental evaluation to successfully reject almost all incorrect predictions. In our experiments on a benchmark suite of six hybrid systems, we found that the frequentist approach consistently outperforms the Bayesian one. We also observed that the Bayesian approach is less practical, requiring a careful and problem-specific choice of hyperparameters.

show abstract