Object and Text-guided Semantics for CNN-based Activity Recognition

Eum, Sungmin; Reale, Christopher; Kwon, Heesung; Bonial, Claire; Voss, Clare R.

doi:10.1109/icassp.2019.8682698

Cited by 5 publications

(2 citation statements)

References 22 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Multiple task-specific layers. Multi-task learning, performing multiple tasks with different layers in a common learned space, is widely used and has presented better accuracy than using multiple single task networks [9], [48]- [51].…”

Section: B Building Cross-domain Pretrained Modelmentioning

confidence: 99%

Exploring Cross-Domain Pretrained Model for Hyperspectral Image Classification

Lee,

Eum,

Kwon

2022

Preprint

Self Cite

View full text Add to dashboard Cite

A pretrain-finetune strategy is widely used to reduce the overfitting that can occur when data is insufficient for CNN training. First few layers of a CNN pretrained on a large-scale RGB dataset are capable of acquiring general image characteristics which are remarkably effective in tasks targeted for different RGB datasets. However, when it comes down to hyperspectral domain where each domain has its unique spectral properties, the pretrain-finetune strategy no longer can be deployed in a conventional way while presenting three major issues: 1) inconsistent spectral characteristics among the domains (e.g., frequency range), 2) inconsistent number of data channels among the domains, and 3) absence of large-scale hyperspectral dataset.We seek to train a universal cross-domain model which can later be deployed for various spectral domains. To achieve, we physically furnish multiple inlets to the model while having a universal portion which is designed to handle the inconsistent spectral characteristics among different domains. Note that only the universal portion is used in the finetune process. This approach naturally enables the learning of our model on multiple domains simultaneously which acts as an effective workaround for the issue of the absence of large-scale dataset.We have carried out a study to extensively compare models that were trained using cross-domain approach with ones trained from scratch. Our approach was found to be superior both in accuracy and in training efficiency. In addition, we have verified that our approach effectively reduces the overfitting issue, enabling us to deepen the model up to 13 layers (from 9) without compromising the accuracy.

show abstract

Section: B Building Cross-domain Pretrained Modelmentioning

confidence: 99%

Exploring Cross-Domain Pretrained Model for Hyperspectral Image Classification

Lee,

Eum,

Kwon

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Examples of colearning algorithms include co-training, zero-shot learning and concept learning. Recent examples include: the use of information extraction in text processing to guide entity detection in computer vision algorithms, such as for visual object recognition, when an implicit instrument argument can be inferred for an event mentioned in a caption accompanying an image (Subburathinam et al, 2019) and multitask learning that exploits a text-guided semantic space to select the most relevant visual objects for novel visual activity recognition (Eum et al, 2019). The challenges that arise in multimodal processing may involve various combinations of these five categories.…”

Section: Levels Of Representation / Abstraction / Granularity / Align...mentioning

confidence: 99%

Report from the NSF Future Directions Workshop, Toward User-Oriented Agents: Research Directions and Challenges

Eskenazi,

Zhao

2020

Preprint

View full text Add to dashboard Cite

This USER Workshop was convened with the goal of defining future research directions for the burgeoning intelligent agent research community and to communicate them to the National Science Foundation. It took place in Pittsburgh Pennsylvania on October 24 and 25, 2019 and was sponsored by National Science Foundation Grant Number IIS-1934222. Any opinions, findings and conclusions or future directions expressed in this document are those of the authors and do not necessarily reflect the views of the National Science Foundation. The 27 participants presented their individual research interests and their personal research goals. In the breakout sessions that followed, the participants defined the main research areas within the domain of intelligent agents and they discussed the major future directions that the research in each area of this domain should take.

show abstract