Multimodal Intelligent Eye-Gaze Tracking System

Abstract

This article presents a series of user studies to develop a new eye-gaze tracking-based pointing system. We developed a new target prediction model that works for different input modalities and combined the eye-gaze tracking-based pointing with a joystick controller that can reduce pointing and selection times. The system finds important applications in cockpit of combat aircraft and for computer novice users. User studies confirmed that users can perform significantly faster using this new eye-gaze tracking-based system for both military and everyday computing tasks compared to existing input devices. As part of the study it was also found that the amplitude of maximum power component obtained through Fourier Transform of pupil signal significantly correlates with selection times and perceived cognitive load of users in terms of Task Load Index scores.

Eye Gaze Controlled Projected Display in Automotive and Military Aviation Environments

“…As per our knowledge, there is not any reported work on dual task studies involving flight simulator and gaze controlled interface. However, the average response time reported in study 4 was even lower than the gaze controlled head down display reported by Biswas [32] in a single task study.…”

“…A previous study [32] explored hardware switch and voice command based selection while researchers on assistive technology [40] explored dwell time based selection. Considering the timing constraint, we used a button on steering and throttle (slew button) for selecting targets in the reported user studies.…”

“…Thomas [31] reported results from an ISO 9241 task involving thumbstick, trackball, touchpad and touchscreen and found the trackpad to be the most accurate and argued for supporting new modalities for the HOTAS Cursor Control Device (CCD). Biswas [32] reported significant reduction in reaction time for gaze controlled interface compared to HOTAS TDS for operating a simulated MFD, but their study did not involve any primary flying task and was configured in a desktop computing environment.…”

“…The pointer works as feedback to the user in case the sensor could not track accurately and the user can make corrective eye gaze or finger movement to select the target. The target prediction system [32] highlights the nearest target to the eye gaze or finger position and a hardware switch push can be used to select the target even if the pointer is not exactly on the target button itself. proposed system records eye gaze positions continuously and takes the median of the pixel locations in every 300 ms to estimate the region of interest or saccadic focus points.…”

“…However, eye gaze movements are not as smooth as mouse or finger or hand movement, rather they follow a 'spotlight' metaphor and, so far, not any next point or target prediction models are tested for gaze controlled interface. A backpropagation neural network model was used in [32], but only to differentiate between ballistic and homing movements. Gaze tracking is used for zooming or accentuating part of display to help pointing by another pointing modality [6,9,22].…”