Multimodal Emotion and Sentiment Modeling From Unstructured Big Data: Challenges, Architecture, &amp; Techniques

Seng, Jasmine Kah Phooi; Ang, Li-Minn

doi:10.1109/access.2019.2926751

Cited by 26 publications

(14 citation statements)

References 79 publications

(95 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This section gives brief discussions and explores the potential of ensemble machine learning and scalable parallel discriminant analysis (SPDA) for agriculture information processing towards the application of hyperspectral image classification. A similar approach to the proposed SPDA has been previously reported for human emotion and sentiment classification from unstructured Big data [69]. However, the potential of ensemble machine learning and scalable parallel discriminant analysis (EML-SPDA) has not been explored in agriculture information processing.…”

Section: Ensemble Machine Learning and Scalable Parallel Discrimmentioning

confidence: 95%

“…This part of the paper discusses the EML-SPDA to address Challenges (1) and 2for Big hyperspectral data for agricultural systems. A difference between the previous work and the proposed approach is that the work in [69] was targeted towards twodimensional facial image data, whereas the proposed approach is targeted towards large volume three-dimensional (3-D) hyperspectral spatial-spectral data cubes (i.e. Big hyperspectral data).…”

Section: Ensemble Machine Learning and Scalable Parallel Discrimmentioning

confidence: 99%

See 1 more Smart Citation

Big Data and Machine Learning With Hyperspectral Information in Agriculture

2021

Self Cite

View full text Add to dashboard Cite

Hyperspectral and multispectral information processing systems and technologies have demonstrated its usefulness for the improvement of agricultural productivity and practices by providing useful information to farmers and crop managers on the factors affecting crop status and growth. These technologies are widely used in a range of agriculture applications such as crop management, crop yield forecasting, crop disease detection, and the monitoring of agriculture land usage, water, and soil conditions. Hyperspectral information sensing can acquire several hundred spectral bands that cover the electromagnetic spectrum of an observational scene in a single acquisition. The resulting hyperspectral data cube contains a large volume of spatial and spectral information. The hyperspectral sequence of images or video further increases the data generation velocity and volume which lead to the Big data challenges particularly in agricultural remote sensing applications. This paper is structured to first give a comprehensive review of representative studies to provide insights into significant research efforts in agriculture using Big data, machine learning and deep learning with the focus on frameworks or architectures, information processing and analytics with hyperspectral and multispectral data. The potential for utilizing Big data, machine learning and deep learning for hyperspectral and multispectral data in agriculture is very promising. The paper then further explores the potential of using ensemble machine learning and scalable parallel discriminant analysis which takes into consideration the spatial and spectral components for Big data in agriculture. To the best of our knowledge, no similar review study on agriculture with Big data, machine learning and deep learning for hyperspectral and multispectral information processing has been reported. Furthermore, the potential of ensemble machine learning and scalable parallel discriminant analysis has not been explored in agriculture information processing. Experiments and data analytics have been performed on hyperspectral data from agriculture for validation. The results have shown the good performance of our approach.

show abstract

Section: Ensemble Machine Learning and Scalable Parallel Discrimmentioning

confidence: 95%

Section: Ensemble Machine Learning and Scalable Parallel Discrimmentioning

confidence: 99%

Big Data and Machine Learning With Hyperspectral Information in Agriculture

2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Tactile-based methods extract features from tactile sensors and are used in human-robot interaction where emotion is estimated based on the types of physical interaction [12]. Multimodal methods aim to improve the overall accuracy of emotion recognition using multiple data sources [13]. A good overview is provided in [14].…”

Section: Related Workmentioning

confidence: 99%

Real-time Emotion Recognition for Sales

Naas

Sigg

2020

2020 16th International Conference on Mobility, Sensing and Networking (MSN)

View full text Add to dashboard Cite

Positive emotion is a pre-condition to any sales contract. Likewise, the ability to perceive the emotions of a customer impacts sales performance.To support emotional perception in buyer-seller interactions, we propose an audio-visual emotion recognition system that can recognize eight emotions: neutral, calm, sad, happy, angry, fearful, surprised, and disgusted. We reduced noise in audio samples and we applied transfer learning for image feature extraction based on a pre-trained deep neural network VGG16. For emotion recognition, we successfully obtained an audio emotion-recognition accuracy of 62.51% and 68% and video emotion-recognition accuracy of 97.13% and 97.77% on the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) and Surrey Audio-Visual Expressed Emotion (SAVEE) datasets respectively. For the combination of the two models, our proposed merging mechanism without re-training achieved an accuracy of close to 100% on both datasets. Finally, we demonstrated our system for a customer satisfaction use case in a real customer-to-salesperson interaction using audio and video models, achieving an average accuracy of 78%.

show abstract

“…Unstructured Big Data content is rapidly growing in social media [9]. Previously text-based, nowadays, social media posts contain images and videos.…”

Section: Introductionmentioning

confidence: 99%

Multiview Sentiment Analysis with Image-Text-Concept Features of Indonesian Social Media Posts

Setiawan¹

2021

IJIES

View full text Add to dashboard Cite

Social media development makes it possible for everyone to express their opinions and information through text, speech, video, or images. Multiview sentiment analysis in current studies generally combines two modalities, text and image. It seeks to classify social media posts into two or more polarities, such as positive, neutral, or negative. To improve the performance of multiview Sentiment Analysis, we added another modality, which is concepts derived from text and image. Our proposed model integrates three views into a fusion with an ensemble approach by a metaclassifier. We performed text classification with Deep Convolutional Neural Networks. The input feature is Word2Vec for text representation in order to preserve semantic meaning. Additionally, we analyzed concepts from texts with SenticNet 5 as a knowledge base model and extracted concepts from images using the DeepSentiBank model. We obtained 2089 Adjective Noun Pairs and classified it with Multi-Layer Perceptron. Then we combined predicted probabilities from each classifier for Image, Text, and Concept by Ensemble Learning. A meta-classifier was implemented to predict the final sentiment from a fusion of Image-Text-Concept features. The fusion for multiview sentiment analysis works well and could achieve the best accuracy of 70% by applying the ensemble approach with Logistic Regression as the meta-classifier.

show abstract

Multimodal Emotion and Sentiment Modeling From Unstructured Big Data: Challenges, Architecture, & Techniques

Cited by 26 publications

References 79 publications

Big Data and Machine Learning With Hyperspectral Information in Agriculture

Big Data and Machine Learning With Hyperspectral Information in Agriculture

Real-time Emotion Recognition for Sales

Multiview Sentiment Analysis with Image-Text-Concept Features of Indonesian Social Media Posts

Contact Info

Product

Resources

About