Aggregating semantic concepts for event representation in lifelogging

Abstract

The performance of automatic detection of concepts in image and video data has been improved to a satisfactory level for some generic concepts like indoor, outdoor, faces, etc. on high quality data from broadcast TV or movies. However it remains a challenge to apply this to interpreting the high-level semantics of events as they occur in visual lifelogs from wearable cameras. This is because poorer quality image data and the activities of the wearer make it difficult to automatically categorise them. In this paper, we propose an interestingness-based semantic aggregation and representation algorithm, to tackle the problem of event management and representation in visual lifelogging. Semantic concept interestingness is calculated by fusing image-level concepts which are then exploited to select a representation for the semantic event correlated to various event topics. Experimental results show the efficacy of our algorithm in fusing semantics at the event level, and in selecting representations for event management in visual lifelogging.

Event-based media processing and analysis: A survey of the literature

“…The goal of lifelogging is to analyze a person's everyday behavior and experiences in terms of events, states, and relationships [200] in order to support and facilitate everyday life. Events, as discrete and repetitive activities, are defined as the unit of interaction in a lifelogging system [200,201]. A challenging problem in a lifelogging system is to detect specific events after identifying first the boundaries of them.…”

“…In [200], the authors tackle this challenge by semantically enriching lifelog events and creating semantic links between those descriptors and other external knowledge. Wang et al [201] proposed an interestingness-based semantic aggregation and representation algorithm, to tackle the problem of event management and representation in visual lifelogging. Semantic concept interestingness is calculated by fusing image-level concepts which are then exploited to select a representation for the semantic event correlated to various event topics.…”

“…Event-based media processing and analysis is currently a hot topic being used in a broad range of scientific and consumer domains, a sample of which include: (a) multimedia organization and consumer video management for digital preservation [46,165,228,42,34], (b) lifelogging [80,200,201,153], (c) video surveillance [169,39,85,101], (d) Journalism (News-and sport-related applications for community awareness) [203,170,146,113,166,204,51], (e) social media [151,161,174,86], and (f) event visualization [48,160,38]. In this paper we provide an extensive overview of the literature in this growing domain.…”

Characterizing everyday activities from visual lifelogs based on enhancing concept representation

“…In visual lifelogging, much work has been done on activity processing such as automatic event segmentation [10], event representation [53], life pattern analysis [23], event enhancement [11] and so on. To drive these applications, all of the work uses images from wearable cameras and some additionally uses other captured metadata such as location from GPS, date and time.…”

“…To tackle the image quality problem introduced by wearers' movements despite the on-board movement sensors on the SenseCam, low-quality images are filtered out according to a fusion of the Contrast and Saliency measures which, in previous work, is shown to be effective in choosing high-quality lifelogging event representations [8,54]. The 23 everyday activity types listed in Table 1 are applied in the evaluation for which 12,248 visual lifelog images are involved in our experiment [56].…”