A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

Abstract

Multimodal machine learning (MML) is a tempting multidisciplinary research area where heterogeneous data from multiple modalities and machine learning (ML) are combined to solve critical problems. Usually, research works use data from a single modality, such as images, audio, text, and signals. However, real-world issues have become critical now, and handling them using multiple modalities of data instead of a single modality can significantly impact finding solutions. ML algorithms play an essential role by tuning parameters in developing MML models. This paper reviews recent advancements in the challenges of MML, namely: representation, translation, alignment, fusion and co-learning, and presents the gaps and challenges. A systematic literature review (SLR) applied to define the progress and trends on those challenges in the MML domain. In total, 1032 articles were examined in this review to extract features like source, domain, application, modality, etc. This research article will help researchers understand the constant state of MML and navigate the selection of future research directions.

The use of EEG and VR in Mental State Recognition

“…Wavy lines, representing the timevarying voltages at each electrode location, can be seen on an EEG recording [5]. EEG recordings of brain waves, or neural signals, are used in brain-computer interfaces (BCI), medicine, sleep studies, and other applications including epilepsy, neurology, cognitive science, and psychophysiological research [6]. The EEG is a non-stationary and nonlinear signal [6].…”

“…EEG recordings of brain waves, or neural signals, are used in brain-computer interfaces (BCI), medicine, sleep studies, and other applications including epilepsy, neurology, cognitive science, and psychophysiological research [6]. The EEG is a non-stationary and nonlinear signal [6]. The EEG signal amplitude is between 10-100 uV.…”

“…EEG noise artifacts can be divided into two categories which are non-physiological and physiological artifacts [6]. Non-physiological artifacts, sometimes referred to as external artifacts, are typically brought on by the device and the recording equipment.…”

“…These artifacts are caused by malfunctions in the device, such as interference from electric fields, faulty electrode connections, interference from surrounding electronic equipment' electromagnetic fields, etc. Power line noise at either 50 or 60 Hz, cable movement, perspiration, and electrode movement are examples of sources of non-physiological aberrations in EEG signals [6] As for physiological artifacts, eye and head movements are frequent sources of physiological artifacts. Muscular artifacts, such as skeletal or facial muscle activity, frequently taint EEG readings, which is an issue.…”

“…EEG signal artifacts can also be caused by cardiac activity. In summary, the origins of physiological artifacts may be divided into three groups: 1) muscle activity (EMG) and 2) ocular activity (EOG) and 3) cardiac activity [6].…”