Uncertainty Evaluation of Temporal Trust in a Fusion System Using the URREF Ontology

Villiers, J. P. de; Pavlin, Gregor; Ziegler, J.; Jousselme, Anne-Laure; Costa, Paulo C. G.; Blasch, E.; Laskey, Kathryn Blackmond; Laudy, Claire; Waal, Alta de; Cho, Judy H.

doi:10.23919/fusion49465.2021.9627007

Cited by 11 publications

(11 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Exp 6 clearly demonstrates the consequences of ignoring the data structure, resulting in huge quantities of training data to achieve close to optimal classification performance. In this case |D train | ′ was accurately predicted by using equation (6) .…”

Section: B Adequate Quantities Of Training Datamentioning

confidence: 96%

“…This paper proposes the concept of Qualitative Models of Data Generating Processes (QM-DGP) that describe causal dependencies between different phenomena influencing the generation of observations in a qualitative manner. The approach is inspired by the principles expressed in [5] and can be viewed as an extension of the Abstraction flow concepts introduced in [6]. In particular, QM-DGP supports modelling of real world entities and processes (RWEPs), the basis for various types of abstractions and analyses throughout a system's life-cycle.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Qualitative Models of Data Generation Processes: Facilitating Data-Intensive AI Solutions

Pavlin,

Laskey,

Mignet

et al. 2023

2023 26th International Conference on Information Fusion (FUSION)

View full text Add to dashboard Cite

AI-based decision support solutions require life cycles that adequately address critical steps, such as (i) finding suitable machine learning (ML) methods for the problem at hand, (ii) preparing and executing adequate data acquisition processes and (iii) tractable evaluation of the overall solution. Understanding the data generating processes is key in achieving this. Training and test data can be seen as a result of a causal data generation process, a sampling process in which the data is collected from different sources that are influenced by multiple interdependent phenomena. This is represented by a Qualitative Model of Data Generation Processes (QM-DGP), a causal graphical model. QM-DGP facilitates analysis of the complexity of the underlying data generating processes that can inform the development of trustable ML-based solutions in multiple ways. Firstly, this analysis is the basis for the determination of the required complexity of the ML models. Secondly, it facilitates the determination of the quantities of training data supporting good learning results. Thirdly, it can provide guidance for a systematic simplification of the models, supporting tractable solutions without significantly reduced performance. The construction of QM-DGP and the analysis benefit from sound theoretical concepts, such as dseparation and I-Maps. Experimental results with simulated data indicate that the approach can be effective in predicting the required quantities of training data and the determination of the modelling complexity using different types of models.

show abstract

Section: B Adequate Quantities Of Training Datamentioning

confidence: 96%

Section: Introductionmentioning

confidence: 99%

Qualitative Models of Data Generation Processes: Facilitating Data-Intensive AI Solutions

Pavlin,

Laskey,

Mignet

et al. 2023

2023 26th International Conference on Information Fusion (FUSION)

View full text Add to dashboard Cite

show abstract

“…Information fusion using evidence theory provides an additional feature: the uncertainty quantification [13]. The uncertainty serves to assess the output reliability of the combined system [34]. Alternatively, knowledge-based models are expert-centered approaches containing valuable expert domain and environment context [35].…”

Section: B Information Fusionmentioning

confidence: 99%

Adaptive Information Fusion Using Evidence Theory and Uncertainty Quantification

Arévalo,

Alison,

Ibrahim

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Fault detection systems support the operator, providing insight during the decision-making while having an (unknown) fault. Data-based models are a common option for a detection system. However, systems that rely purely on data-based models are normally trained with a specific set of data, which cannot necessarily prevent data drift. Thus, an anomaly or unknown condition detection mechanism is required to handle data with new fault cases. Besides, the model's capability to adapt to the unknown condition is equally important to anomaly detection-in other words, its capability to update itself automatically. Alternatively, expert-centered models are powered by the knowledge of operators, which provides the models with production context and expert domain knowledge. The challenge lies in combining both systems and which framework can be used to achieve this fusion. We propose an adaptive information fusion methodology to define fault detection systems using evidence theory and uncertainty quantification. The main contribution of this paper is providing a general framework for the fusion of n number of information sources using the evidence theory. The fusion provides a more robust prediction and an associated uncertainty that can be used to assess the prediction likeliness. Moreover, we provide a methodology for the information fusion of two primary sources: an ensemble classifier based on machine data and an expert-centered model. We demonstrate the information fusion approach using data from an industrial setup, which rounds up the application part of this research. Furthermore, we address the problem of data drift by proposing a methodology to update the data-based models using an evidence theory approach. We validate the approach using the Benchmark Tennessee Eastman while doing an ablation study of the model update parameters.

show abstract

“…Adversarial noise in this instance refers to noise that exploits vulnerabilities in an ML system. In [30], a taxonomy for uncertainty representation and evaluation for modelling and decision-making in information fusion is presented and is further extended in [31]. It contains a discussion on the different types of uncertainties and where they enter a sensing or fusion system.…”

Section: Related Work 121 Comparison Of Atr Systemsmentioning

confidence: 99%

Feedback-Assisted Automatic Target and Clutter Discrimination Using a Bayesian Convolutional Neural Network for Improved Explainability in SAR Applications

et al. 2022

View full text Add to dashboard Cite

In this paper, a feedback training approach for efficiently dealing with distribution shift in synthetic aperture radar target detection using a Bayesian convolutional neural network is proposed. After training the network on in-distribution data, it is tested on out-of-distribution data. Samples that are classified incorrectly with high certainty are fed back for a second round of training. This results in the reduction of false positives in the out-of-distribution dataset. False positive target detections challenge human attention, sensor resource management, and mission engagement. In these types of applications, a reduction in false positives thus often takes precedence over target detection and classification performance. The classifier is used to discriminate the targets from the clutter and to classify the target type in a single step as opposed to the traditional approach of having a sequential chain of functions for target detection and localisation before the machine learning algorithm. Another aspect of automated synthetic aperture radar detection and recognition problems addressed here is the fact that human users of the output of traditional classification systems are presented with decisions made by “black box” algorithms. Consequently, the decisions are not explainable, even to an expert in the sensor domain. This paper makes use of the concept of explainable artificial intelligence via uncertainty heat maps that are overlaid onto synthetic aperture radar imagery to furnish the user with additional information about classification decisions. These uncertainty heat maps facilitate trust in the machine learning algorithm and are derived from the uncertainty estimates of the classifications from the Bayesian convolutional neural network. These uncertainty overlays further enhance the users’ ability to interpret the reasons why certain decisions were made by the algorithm. Further, it is demonstrated that feeding back the high-certainty, incorrectly classified out-of-distribution data results in an average improvement in detection performance and a reduction in uncertainty for all synthetic aperture radar images processed. Compared to the baseline method, an improvement in recall of 11.8%, and a reduction in the false positive rate of 7.08% were demonstrated using the Feedback-assisted Bayesian Convolutional Neural Network or FaBCNN.

show abstract

Uncertainty Evaluation of Temporal Trust in a Fusion System Using the URREF Ontology

Cited by 11 publications

References 16 publications

Qualitative Models of Data Generation Processes: Facilitating Data-Intensive AI Solutions

Qualitative Models of Data Generation Processes: Facilitating Data-Intensive AI Solutions

Adaptive Information Fusion Using Evidence Theory and Uncertainty Quantification

Feedback-Assisted Automatic Target and Clutter Discrimination Using a Bayesian Convolutional Neural Network for Improved Explainability in SAR Applications

Contact Info

Product

Resources

About