Discovering informative social subgraphs and predicting pairwise relationships from group photos

Abstract

An increasing number of users are contributing the sheer amount of group photos (e.g., for family, classmates, colleagues, etc.) on social media for the purpose of photo sharing and social communication. There arise strong needs for automatically understanding the group types (e.g., family vs. classmates) for recommendation services (e.g., recommending a family-friendly restaurant) and even predicting the pairwise relationships (e.g., mother-child) between the people in the photo for mining implicit social connections. Interestingly, we observe that the group photos are composed of atomic subgroups corresponding to certain social relationships. For this work, we propose a novel framework to (1) connect faces of different attributes and positions as a face graph and (2) discover informative subgraphs to represent social subgroups in group photos. A group photo can be further represented by a bag-of-face-subgraphs (BoFG) -the occurring frequency of social subgroups, which is informative to categorize specific group types or events. We demonstrate the effectiveness of BoFG in recognizing family photos and achieve 30.5% relative improvement over the state-of-the-art low-level features. Moreover, we propose to predict the pairwise relationships (e.g., husband-wife) in a face graph by the co-occurrence information (e.g., co-occurring with a child) in the mined subgraphs. The experiments demonstrate that the informative social subgroups significantly outperform prior work (36% relatively) which considers merely facial attributes for determining pairwise relationships. Categories and Subject Descriptors I.4 [Image Processing and Computer Vision]: Feature Measurement General Terms Algorithms, Experimentation, Human Factors KeywordsFace graph, face subgraph mining, social relationship Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MM'12, October 29-November 2, 2012, Nara, Japan.

Deciphering the Crowd: Modeling and Identification of Pedestrian Group Motion

“…Several works address investigation of social relations based on such universally valid implicit cues as the age difference between parents and children or the opposite genders of heterosexual couples [10,11]. Some studies investigate kin relationships using photo albums that span a long time window of several years or even decades [12,13].…”

Toward Personalised and Dynamic Cultural Routing: A Three-Level Approach

“…Thus, Cheng et al (2011) analysed a large pool of public travel photos taken by tourists at a number of popular destinations and applied a face detection technique to extract from those photos certain details about the tourists (namely, their age, gender, and race), which allowed them to discover travel preferences of different categories of tourists and to build a tourist recommendation model 6 . In a follow-up study 8 , they showed how social relationships could also be identified from analysing group photos, which could be applied to personalise group recommendation services. In another example, Tsai & Chung (2012) leveraged the use of the RFID technology to track the behaviour of a theme park's visitors for identifying route patterns that were matched to the input the visitors provided, so that a route recommendation system was built that suggested the newly arrived visitors the most suitable visit programme based on the experience of previous visitors with similar profiles 7 .…”

3D Visual Proxemics: Recognizing Human Interactions in 3D from a Single Image

“…There are two main directions in this line of work -2D approaches that directly translate image based features into concepts, and 3D approaches that translate detected faces into the 3D scene and then derive features from the 3D layout. Examples of 2D approaches include [5] in which authors predict pairwise relationships e.g., couple, sibling etc. from facial attributes and face subgraphs, and [19], in which authors label pairs of people with a physically-based touch codes such as Hand-hand, Hand-torso etc.…”

“…A number of research groups [19,5,10,9] have conducted insightful studies for understanding people interactions in images and videos, though with limited scope. Most of these approaches [19,5] perform their analysis in the 2D camera space.…”

“…Most of these approaches [19,5] perform their analysis in the 2D camera space. Although these approaches demonstrated their effectiveness, their robustness is fundamentally limited by the 2D analysis paradigm and cannot handle the diversity in camera pose and people depths often seen in user generated Internet content.…”