BabyTalk: Understanding and Generating Simple Image Descriptions

Abstract

Abstract-We present a system to automatically generate natural language descriptions from images. This system consists of two parts. The first part, content planning, smooths the output of computer vision-based detection and recognition algorithms with statistics mined from large pools of visually descriptive text to determine the best content words to use to describe an image. The second step, surface realization, chooses words to construct natural language sentences based on the predicted content and general statistics from natural language. We present multiple approaches for the surface realization step and evaluate each using automatic measures of similarity to human generated reference descriptions. We also collect forced choice human evaluations between descriptions from the proposed generation system and descriptions from competing approaches. The proposed system is very effective at producing relevant sentences for images. It also generates descriptions that are notably more true to the specific image content than previous work.

Interleaved text/image Deep Mining on a large-scale radiology database

“…Our work has been inspired by the works building very large-scale image databases [8,38] and the works establishing semantic connections of texts and images [25]. We observe good semantic coherence between labels obtained by hierarchical document topic models [6] and clinician's assessment.…”

“…Image-to-language correspondence was learned from ImageNet dataset and reasonably high quality image description datasets (Pascal1K [36], Flickr8K [16], Flickr30K [47]) in [20], where such caption datasets are not available in the medical domain. Graphical models have been employed to predict image attributes ( [27,39]), or to describe images ( [25]) using manually annotated datasets ( [36,26]). Automatic label mining on large, unlabeled datasets is presented in [35,18], however the variety of the label-space is limited (image text annotations).…”

“…Our deep CNN models on medical image modalities (mostly CT, MRI) are initialized with the model parameters pre-trained from ImageNet [8] using Caffe [19] framework. Kulkarni et al [25] have spearheaded the efforts of learning the semantic connections between image contents and the sentences describing them (i.e., captions). Detecting objects of interest, attributes and prepositions and applying contextual regularization with a conditional random field (CRF) is a feasible approach [25] because many useful tools are available in computer vision.…”

“…Kulkarni et al [25] have spearheaded the efforts of learning the semantic connections between image contents and the sentences describing them (i.e., captions). Detecting objects of interest, attributes and prepositions and applying contextual regularization with a conditional random field (CRF) is a feasible approach [25] because many useful tools are available in computer vision. There has not yet been much comparable development on large-scale medical imaging understanding.…”

The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding

“…Here we take an analagous approach-modifying the image retrieval stage of data-driven pipeline-for the task of image captioning. There has been significant recent interest in generating natural language descriptions of photographs (Kulkarni et al 2013;Farhadi et al 2010b). These techniques are typically quite complex: they recognize various visual concepts such as objects, materials, scene types, and the spatial relationship among these entities, and then generate plausible natural language sentences based on this scene understanding.…”