Event-Driven Semantic Concept Discovery by Exploiting Weakly Tagged Internet Images

Abstract-Traditional approaches for classifying event videos rely on a manually curated training dataset. While this paradigm has achieved excellent results on benchmarks such as TrecVid multimedia event detection (MED) challenge datasets, it is restricted by the effort involved in careful annotation. Recent approaches have attempted to address the need for annotation by automatically extracting images from the web, or generating queries to retrieve videos. In the former case, they fail to exploit additional cues provided by video data, while in the latter, they still require some manual annotation to generate relevant queries. We take an alternate approach in this paper, leveraging the synergy between visual video data and the associated textual metadata, to learn event classifiers without manually annotating any videos. Specifically, we first collect a video dataset with queries constructed automatically from textual description of events, prune irrelevant videos with text and video data, and then learn the corresponding event classifiers. We evaluate this approach in the challenging setting where no manually annotated training set is available, i.e., EK0 in the TrecVid challenge, and show state-of-the-art results on MED 2011 and 2013 datasets.

show abstract

“…Works such as [3], [7], [9], [25] learn a model to detect events 1 The '0' denotes the number of training examples provided.…”

Section: Related Workmentioning

confidence: 99%

Learning From Web Videos for Event Classification

Chesneau

Alahari

Schmid

2018

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…Second, mid-level concepts are basic actions, activities or interactions. Actions or activities are a "sequence of movements" [Chen et al 2014] and can be performed by one entity, such as people or objects. Interactions are actions between two or more entities.…”

Section: Vocabularymentioning

confidence: 99%

“…Third, highlevel concepts are "complex activities that involve people interacting with other people and/or objects under certain scene" [Chen et al 2014]. The key difference between midlevel and high-level concepts is that a high-level concept contains multiple actions and interactions evolving over time [Chen et al 2014], such as the difference between the action horse riding and the event horse riding competition. Furthermore, concepts can have different levels of granularity, also referred to as specificity.…”

Section: Vocabularymentioning

confidence: 99%

“…In this paper, we show the importance of high-level concepts, defined as "complex activities that involve people interacting with other people and/or objects under certain scene" [Chen et al 2014], because a combination of objects and actions often cannot capture the full semantics of a high-level event. We do not claim that we are the first to use high-level concepts, but we show the difference in performance for different types of concepts.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Semantic Reasoning in Zero Example Video Event Retrieval

Boer

Zhang

et al. 2017

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

Searching in digital video data for high-level events, such as a parade or a car accident, is challenging when the query is textual and lacks visual example images or videos. Current research in deep neural networks is highly beneficial for the retrieval of high-level events using visual examples, but without examples it is still hard to 1) determine which concepts are useful to pre-train (Vocabulary challenge); 2) which pre-trained concept detectors are relevant for a certain unseen high-level event (Concept Selection challenge). In our paper, we present our Semantic Event Retrieval System which 1) shows the importance of high-level concepts in a vocabulary for the retrieval of complex and generic high-level events and 2) uses a novel concept selection method (i-w2v) based on semantic embeddings. Our experiments on the international TRECVID Multimedia Event Detection benchmark show that a diverse vocabulary including high-level concepts improves performance on the retrieval of high-level events in videos and that our novel method outperforms a knowledge-based concept selection method.

show abstract

“…One approach to deal with this challenge is to use Flickr. Both Leong et al [22] and Chen et al [10] use Flickr to find 'picturable' words by using the co-occurrance of tags provided with the images resulting from a query. ConceptNet [44] has high potential, but it has not yet shown significant improvement of performance in finding a known item [50].…”

Section: Complex Event Detectionmentioning

confidence: 99%

Knowledge based query expansion in complex multimedia event detection

Boer

Schutte

Kraaij

2015

Multimed Tools Appl

View full text Add to dashboard Cite

A common approach in content based video information retrieval is to perform automatic shot annotation with semantic labels using pre-trained classifiers. The visual vocabulary of state-of-the-art automatic annotation systems is limited to a few thousand concepts, which creates a semantic gap between the semantic labels and the natural language query. One of the methods to bridge this semantic gap is to expand the original user query using knowledge bases. Both common knowledge bases such as Wikipedia and expert knowledge bases such as a manually created ontology can be used to bridge the semantic gap. Expert knowledge bases have highest performance, but are only available in closed domains. Only in closed domains all necessary information, including structure and disambiguation, can be made available in a knowledge base. Common knowledge bases are often used in open domain, because it covers a lot of general information. In this research, query expansion using common knowledge bases ConceptNet and Wikipedia is compared to an expert description of the topic applied to content-based information retrieval of complex events. We run experiments on the Test Set of TRECVID MED 2014. Results show that 1) Query Expansion can improve performance compared to using no query expansion in the case that the main noun of the query could not be matched to a concept detector; 2) Query expansion using expert knowledge is not necessarily better than query expansion using common knowledge; 3) ConceptNet performs slightly better than Wikipedia; 4) Late fusion can slightly improve performance. To conclude, query expansion has potential in complex event detection.

show abstract

Event-Driven Semantic Concept Discovery by Exploiting Weakly Tagged Internet Images

Cited by 69 publications

References 38 publications

Learning From Web Videos for Event Classification

Learning From Web Videos for Event Classification

Semantic Reasoning in Zero Example Video Event Retrieval

Knowledge based query expansion in complex multimedia event detection

Contact Info

Product

Resources

About