Semantic analysis and retrieval in personal and social photo collections

User evaluation of automatically generated keywords and toponyms for geo‐referenced images

“…Annotation itself can simply be considered to be (a) the generation of features that can be indexed for retrieval purpose, for example, through consideration of text associated with an image, (b) features directly extracted from image content, or (c) automatic annotation based on some form of contextual information associate with the image (Zhang, Islam, & Lu, ). As discussed above, the first two approaches can generate indexing features, but these features often do not match the terms used by those seeking images; in other words, they fall victim to the semantic gap (Sandhaus & Boll, ; Smeulders et al., ). In the following text, we therefore first review literature exploring how images are annotated, with the aim of identifying appropriate indexing terms.…”

“…Central to the development of improved annotations, and thus higher quality search, is evaluation of the annotation generated. As explained by Sandhaus and Boll , p. 20), researchers have often established their own test collections, not least because “the most useful data sets are usually designed for one specific research problem,” and Zhang et al. , p. 359) state that “there is no commonly acceptable image database for automatic image annotation.” Typical evaluations have often focused on measures derived from information retrieval such as precision and recall (Müller, Müller, Squire, Marchand‐Maillet, & Pun, ), which though appropriate to summarizing retrieval results, provide no information about why particular annotations are more or less appropriate.…”

“…The importance of individuals and their motivations behind image annotations is termed “top‐down” by Sandhaus and Boll (). They attempt to distinguish such an approach focusing on how humans understand images from approaches that aim to bridge to semantic gap by deriving high‐level semantics directly from individual image features (which they term a “bottom‐up” approach).…”

“…The rapid growth of consumer devices equipped with cameras and GPS and the corresponding growth of georeferenced images indicate the possibility of automatic generation of annotations based on the location of an image, since this appears to be a meaningful way in which individuals index images. GPS coordinates can be considered to be a form of external context (Sandhaus & Boll, ) identified as a potential means to addressing the semantic gap by Smeulders et al. ().…”

Signature-image-based event analysis for personal photo albums

“…This is because of the big difference between human vision system and computer vision w.r.t understanding well the semantic meaning conveyed by images. Although content-based media retrieval has been developed significantly these days, there is still lack of successfully automatic tools to support systems as well as users to control their data [5] [7]. Therefore, we would like to propose a new method to meet such an emerging requirement.…”