Unsupervised Upstream Fusion of Multiple Sensing Modalities Using Dynamic Deep Directional-Unit Networks for Event Behavior Characterization

Garagic, Denis; Pless, Greg Von; Hagan, R; Liu, Fang; Peskoe, Jacob; Zulch, Peter; Rhodes, Bradley J.

doi:10.1109/aero.2019.8742221

Cited by 6 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [ 23 ], support vector machine (SVM) is used as a final method of classification to achieve sense-and-avoid for unmanned aircraft. The use of an autoencoder-based dynamic deep directional unit network [ 24 ] was capable of learning compact and abstract feature representations from high-dimensional spatiotemporal data of full motion video and I/Q data for the purposes of event behavior characterization. Other research into achieving EO/RF fusion for vehicle tracking and detection using Full Motion Video and P-RF includes joint manifold learning [ 25 ], a sheaf-based approach with its data [ 26 ], and SVM classifier [ 23 ].…”

Section: Literature Reviewmentioning

confidence: 99%

“…Combining the two modalities improves overall reliability, and has previously been implemented in a few applications for target detection, estimation, and tracking. While P-RF data have been used in a fusion-focused approach [ 24 , 27 , 28 ], if a blackbox based approach is utilized it can be difficult to understand exactly how the data are used. For this reason, having a level of transparency can be extremely helpful, which is why the main focus of the present research is on explainability with respect to how the modalities are used.…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Finding Explanations in AI Fusion of Electro-Optical/Passive Radio-Frequency Data

Vakil

Blasch

Ewing

et al. 2023

Sensors

View full text Add to dashboard Cite

In the Information Age, the widespread usage of blackbox algorithms makes it difficult to understand how data is used. The practice of sensor fusion to achieve results is widespread, as there are many tools to further improve the robustness and performance of a model. In this study, we demonstrate the utilization of a Long Short-Term Memory (LSTM-CCA) model for the fusion of Passive RF (P-RF) and Electro-Optical (EO) data in order to gain insights into how P-RF data are utilized. The P-RF data are constructed from the in-phase and quadrature component (I/Q) data processed via histograms, and are combined with enhanced EO data via dense optical flow (DOF). The preprocessed data are then used as training data with an LSTM-CCA model in order to achieve object detection and tracking. In order to determine the impact of the different data inputs, a greedy algorithm (explainX.ai) is implemented to determine the weight and impact of the canonical variates provided to the fusion model on a scenario-by-scenario basis. This research introduces an explainable LSTM-CCA framework for P-RF and EO sensor fusion, providing novel insights into the sensor fusion process that can assist in the detection and differentiation of targets and help decision-makers to determine the weights for each input.

show abstract