Inferring generic activities and events from image content and bags of geo-tags

Abstract

The use of contextual information in building concept detectors for digital media has caught the attention of the multimedia community in the recent years. Generally speaking, any information extracted from image headers or tags, or from large collections of related images and used at classification time, can be considered as contextual. Such information, being discriminative in its own right, when combined with pure contentbased detection systems using pixel information, can improve the overall recognition performance significantly. In this paper, we describe a framework for probabilistically modeling geographical information using a Geographical Information Systems (GIS) database for event and activity recognition in general-purpose consumer images, such as those obtained from Flickr. The proposed framework discriminatively models the statistical saliency of geo-tags in describing an activity or event. Our work leverages the inherent patterns of association between events and their geographical venues. We use descriptions of small local neighborhoods to form bags of geo-tags as our representation. Statistical coherence is observed in such descriptions across a wide range of event classes and across many different users. In order to test our approach, we identify certain classes of activities and events wherein people commonly participate and take pictures. Images and corresponding metadata, for the identified events and activities, are obtained from Flickr. We employ visual detectors obtained from Columbia University (Columbia 374), which perform pure visual event and activity recognition. In our experiments, we present the performance advantage obtained by combining contextual GPS information with pixel-based detection systems.

[Invited Paper] A Review of Web Image Mining

“…It is a just 2-dimensional vector. To convert a 2-d vector into richer representation, Luo et al 83) and Yaegashi et al 137) converted geotags into visual information from the sky using aerial images, and Joshi et al 57) transformed geotags to words using reverse geo-coding technique. On the other hand, Yuan et al 150) used GPS trace data which is a series of geotags instead of using just a pinpoint geotag in order to classify images into several pre-defined events.…”

“…Some work defined it as an activity in which people participate and take pictures such as hiking, playing sports at park and wedding party 57) , while in the TRECVID Multimedia Event Detection task it was defined as an abstract concept of "action" or complex actions, and includes more personal activities such as making a sandwich, repairing an appliance and marriage proposal 112) . As another work on activity events, anomaly detection from video/image streams has been studied before 153) .…”

A Survey of Geo-tagged Multimedia Content Analysis within Flickr

“…They observed that by incorporating temporal information, the accuracy of the results was significantly improved. Joshi and Luo [22] used visual detectors and incorporate bags-of-geotags within a probabilistic framework, in order to detect activities and events in photos. Yu and Luo [23] combined visual context with location information in order to detect concepts in photos.…”

Geo-Temporal Mining and Searching of Events from Web-based Image Collections

“…Finally other applications based on the geographical knowledge extracted from Web resources are trajectory mining and travel Itinerary mining for tourist purposes [40,126] and activity recommendation [74]. In addition, Silva and Martins [159] developed a tag suggestion method based on analysis of collective knowledge, spatial proximity extracted from picture collections and automatic assignment of geographical location to a tagged picture.…”