Adaptive Neuro-Fuzzy Fusion of Multi-Sensor Data for Monitoring a Pilot’s Workload Condition

Abstract

To realize an early warning of unbalanced workload in the aircraft cockpit, it is required to monitor the pilot’s real-time workload condition. For the purpose of building the mapping relationship from physiological and flight data to workload, a multi-source data fusion model is proposed based on a fuzzy neural network, mainly structured using a principal components extraction layer, fuzzification layer, fuzzy rules matching layer, and normalization layer. Aiming at the high coupling characteristic variables contributing to workload, principal component analysis reconstructs the feature data by reducing its dimension. Considering the uncertainty for a single variable to reflect overall workload, a fuzzy membership function and fuzzy control rules are defined to abstract the inference process. An error feedforward algorithm based on gradient descent is utilized for parameter learning. Convergence speed and accuracy can be adjusted by controlling the gradient descent rate and error tolerance threshold. Combined with takeoff and initial climbing tasks of a Boeing 737–800 aircraft, crucial performance indicators—including pitch angle, heading, and airspeed—as well as physiological indicators—including electrocardiogram (ECG), respiration, and eye movements—were featured. The mapping relationship between multi-source data and the comprehensive workload level synthesized using the NASA task load index was established. Experimental results revealed that the predicted workload corresponding to different flight phases and difficulty levels showed clear distinctions, thereby proving the validity of data fusion.

An Interval Type-3 Fuzzy–Fractal Approach for Plant Monitoring

“…Recently, we have seen the utilization of neural networks, evolutionary computing, and fuzzy systems in this area. In the particular case of fuzzy logic, we can find that most of the works in the literature for monitoring are based on the simplest form of fuzzy logic [1][2][3], which is called type-1, like the works that can be reviewed in [4][5][6][7][8][9][10][11][12]. More recently, type-2 has also been considered in this area, as a way to model uncertainty in a better fashion and achieve better results, as can be verified in [13,14].…”

“…In this case, fractal theory constructs are utilized to analyze the complexity of monitoring data to find hidden structure in the data. In addition, hybrid approaches for monitoring have also been proposed, like: genetic fuzzy, neuro-fuzzy, fuzzy-fractal, and others, as in [12][13][14]. In these hybrids, the idea is taking advantage of learning and optimization, provided by other techniques (genetic and neural algorithms), to improve the performance of fuzzy models.…”

Real-time Driver Drowsiness Detection Techniques using Machine Learning Algorithms and Models

“…Zhang and Zhang (2019) [4] explored the fusion of multiple sensors, including EEG and eye-tracking data, to enhance detection accuracy. This paper represents an important contribution to the field by illustrating how multi-sensor integration can provide a more comprehensive understanding of a driver's state, thus improving the reliability of drowsiness detection systems.…”

Temperature Sequential Data Fusion Algorithm Based on Cluster Hierarchical Sensor Networks

“…Turan [11], to overcome the attitude positioning problem of an endoscopic capsule robot, proposed a multi-sensor data fusion method based on a PF and a recurrent neural network. Zhang [12] designed a multi-source data fusion model based on a fuzzy neural network to monitor the real-time workload of pilots and facilitate the timely warning of workload imbalance in the cockpit. The performance of this method was found to be better than those of other data fusion methods.…”