Multi-Modal Deep Analysis for Multimedia

Zhu, Wenwu; Wang, Xin; Zhang, Lipeng

doi:10.1109/tcsvt.2019.2940647

Cited by 28 publications

(4 citation statements)

References 134 publications

(155 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The closer consumers are to the pandemic-affected region/province or city, the more frequently they search to obtain information through convenient, accessible social media, likely to alleviate the fear of the unknown heightened by the information asymmetry. The multi-modal presentations on social media platforms [49] that are visually and audibly stimulating, the feature that "everyone has a microphone", and the possibility that information is untrue pose a monitoring challenge and could mislead the social media audience. The results show that social media use is a path for regional/province epidemics to affect consumer perceptions of the safety of online food purchases, supporting Hypothesis 3.…”

Section: Regional Effectmentioning

confidence: 99%

The Impact of Regional COVID-19 Outbreak on Consumers’ Risk Perception of Purchasing Food Online

2023

View full text Add to dashboard Cite

This paper examines the perception of risk associated with the presence of coronavirus in food purchased online and online vs. offline food shopping during the COVID-19 epidemic. The influence of COVID-19 status on risk perception was tested using the data collected from 742 consumers between December 2021 and January 2022. The empirical approach distinguished between the epidemic’s status in a province (or region), city, and other areas of the country and applied the ordered logit technique. The regional and citywide epidemic increased the perception that online purchases carry the virus and are riskier than those made offline. Further examination showed that the regional/provincial epidemic created the perception that packaging or social media use were risk factors when purchasing food online. Heterogeneity analysis showed that risk perception was significantly higher in affected cities than in non-affected provinces or other provinces. Risk perception differed across five online food categories, with the highest levels for online-ordered meals and fresh products. Strengthening COVID-19 prevention and control in cities and the province, managing risk due to the handling of food purchased online, and government monitoring of social media use will lessen consumers’ risk perceptions and encourage the use of online food offers during epidemics.

show abstract

Section: Regional Effectmentioning

confidence: 99%

The Impact of Regional COVID-19 Outbreak on Consumers’ Risk Perception of Purchasing Food Online

2023

View full text Add to dashboard Cite

show abstract

“…Many works [19], [20], [24], [43]- [46] consider aggregating information from multi-modal data for various image and video content analysis tasks, such as video action recognition, video anomaly detection and localization, and so on. For example, Liu et al [19] and Dan et al [20] used the skeleton data and RGB frames for action recognition.…”

Section: Multi-modal Fusionmentioning

confidence: 99%

Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition

Shu¹,

Yang²,

Yan³

et al. 2021

Preprint

View full text Add to dashboard Cite

This work focuses on the task of elderly activity recognition, which is a challenging task due to the existence of individual actions and human-object interactions in elderly activities. Thus, we attempt to effectively aggregate the discriminative information of actions and interactions from both RGB videos and skeleton sequences by attentively fusing multimodal features. Recently, some nonlinear multi-modal fusion approaches are proposed by utilizing nonlinear attention mechanism that is extended from Squeeze-and-Excitation Networks (SENet). Inspired by this, we propose a novel Expansion-Squeeze-Excitation Fusion Network (ESE-FN) to effectively address the problem of elderly activity recognition, which learns modal and channel-wise Expansion-Squeeze-Excitation (ESE) attentions for attentively fusing the multi-modal features in the modal and channel-wise ways. Specifically, ESE-FN firstly implements the modal-wise fusion with the Modal-wise ESE Attention (M-ESEA) to aggregate discriminative information in modal-wise way, and then implements the channel-wise fusion with the Channelwise ESE Attention (C-ESEA) to aggregate the multi-channel discriminative information in channel-wise way (referring to Figure 1). Furthermore, we design a new Multi-modal Loss (ML) to keep the consistency between the single-modal features and the fused multi-modal features by adding the penalty of difference between the minimum prediction losses on single modalities and the prediction loss on the fused modality. Finally, we conduct experiments on a largest-scale elderly activity dataset, i.e., ETRI-Activity3D (including 110,000+ videos, and 50+ categories), to demonstrate that the proposed ESE-FN achieves the best accuracy compared with the state-of-the-art methods. In addition, more extensive experimental results show that the proposed ESE-FN is also comparable to the other methods in terms of normal action recognition task.

show abstract

“…Formally, synsets refer to sets of instances having the same meanings. The synset induction task plays an important role in the domain of multimodal machine learning [1][2][3]. Take the image captioning task [4][5][6] as an example, in which the machine algorithm attempts to generate a descriptive sentence for a given image.…”

Section: Introductionmentioning

confidence: 99%

A Visually Enhanced Neural Encoder for Synset Induction

Chen,

Feng,

Zhang

et al. 2023

Electronics

View full text Add to dashboard Cite

The synset induction task is to automatically cluster semantically identical instances, which are often represented by texts and images. Previous works mainly consider textual parts, while ignoring the visual counterparts. However, how to effectively employ the visual information to enhance the semantic representation for the synset induction is challenging. In this paper, we propose a Visually Enhanced NeUral Encoder (i.e., VENUE) to learn a multimodal representation for the synset induction task. The key insight lies in how to construct multimodal representations through intra-modal and inter-modal interactions among images and text. Specifically, we first design the visual interaction module through the attention mechanism to capture the correlation among images. To obtain the multi-granularity textual representations, we fuse the pre-trained tags and word embeddings. Second, we design a masking module to filter out weakly relevant visual information. Third, we present a gating module to adaptively regulate the modalities’ contributions to semantics. A triplet loss is adopted to train the VENUE encoder for learning discriminative multimodal representations. Then, we perform clustering algorithms on the obtained representations to induce synsets. To verify our approach, we collect a multimodal dataset, i.e., MMAI-Synset, and conduct extensive experiments. The experimental results demonstrate that our method outperforms strong baselines on three groups of evaluation metrics.

show abstract

Multi-Modal Deep Analysis for Multimedia

Cited by 28 publications

References 134 publications

The Impact of Regional COVID-19 Outbreak on Consumers’ Risk Perception of Purchasing Food Online

The Impact of Regional COVID-19 Outbreak on Consumers’ Risk Perception of Purchasing Food Online

Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition

A Visually Enhanced Neural Encoder for Synset Induction

Contact Info

Product

Resources

About