No abstract

We introduce the first method for automatic image generation from scene-level freehand sketches. Our model allows for controllable image generation by specifying the synthesis goal via freehand sketches. The key contribution is an attribute vector bridged Generative Adversarial Network called EdgeGAN, which supports high visual-quality object-level image content generation without using freehand sketches as training data. We have built a largescale composite dataset called SketchyCOCO to support and evaluate the solution. We validate our approach on the tasks of both object-level and scene-level image generation on SketchyCOCO. Through quantitative, qualitative results, human evaluation and ablation studies, we demonstrate the method's capacity to generate realistic complex scene-level images from various freehand sketches.

It is a sunny day.It is a family picnic. There are four people, a basket, two apples, one cup,two bananas, and on a picnic rug. There is two trees in the distance.Abstract. We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of advancing research on sketch understanding at both the object and scene level. The dataset is created through a novel and carefully designed crowdsourcing pipeline, enabling users to efficiently generate large quantities of realistic and diverse scene sketches. SketchyScene contains more than 29,000 scene-level sketches, 7,000+ pairs of scene templates and photos, and 11,000+ object sketches. All objects in the scene sketches have ground-truth semantic and instance masks. The dataset is also highly scalable and extensible, easily allowing augmenting and/or changing scene composition. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches and showing how the new dataset enables several applications including image retrieval, sketch colorization, editing, and captioning, etc. The dataset and code can be found at https://github.com/SketchyScene/SketchyScene.

It's a moonlit night. There is a moon in the sky. A house is in the middle. A car is in front of the house. Three trees are on the left of the house. Two trees are on the right back of the car. There is a road.It is a moonlit night. There is a yellow moon in the sky. An orange house with light gray roof is in the middle. A yellow car with blue window is in front of the house. Three green trees are on the left of the house. Two green trees are on the right back of the car. There is a yellow road. The sky is blue and all things are on green grass.It is a moonlit night. There is a purple moon in the sky. A yellow house with yellow roof is in the middle. A blue car with light gray window is in front of the house. Three yellow trees are on the left of the house. Two yellow trees are on the right back of the car. There is a light brown road. The sky is gray and all things are on yellow grass.It is a moonlit night. There is a yellow moon in the sky. A blue house with blue roof is in the middle. A red car with blue window is in front of the house. Three green trees are on the left of the house. Two green trees are on the right back of the car. There is a dark gray road. The sky is blue and all things are on green grass. Fig. 1. We present LUCSS, a language-based interactive colorization system for scene sketches. It takes advantage of both instance level segmentation and language models in a unified generative adversarial network, allowing users to accomplish different colorization goals in a form of language instructions. Left: input scene sketch and its content description automatically generated by LUCSS. Three right columns show the colorization results generated by LUCSS following three different instructions at the bottom. Texts underlined are user-specified, with the target colors highlighted in bold.Abstract. We introduce LUCSS, a language-based system for interactive colorization of scene sketches, based on their semantic understanding. LUCSS is built upon deep neural networks trained via a large-scale repository of scene sketches and cartoon-style color images with text descriptions. It consists of three sequential modules. First, given a scene sketch, the segmentation module automatically partitions an input sketch into individual object instances. Next, the captioning module generates the text description with spatial relationships based on the instance-level segmentation results. Finally, the interactive colorization module allows users to edit the caption and produce colored images based on the altered caption. Our experiments show the effectiveness of our approach and the desirability of its components to alternative choices.

No abstract

Image-based indoor localization has aroused much interest recently because it requires no infrastructure support. Previous approaches on image-based localization, due to their computation and storage requirements, often process queries at servers. This does not scale well, incurs round-trip delay, and requires constant network connectivity. Many also require users to manually confirm the shortlisted matched landmarks, which is inconvenient, slow, and prone to selection error. To overcome these limitations, we propose a h ighly a utomated (in terms of image confirmation after taking images) i mage-based l ocalization algorithm (HAIL), distributed in mobile devices. HAIL achieves resource efficiency (in terms of storage and processing) by keeping only distinguishing visual features for each landmark, and employing the efficient k-d tree to search for features. It further utilizes motion sensors and map constraints to enhance the localization accuracy without user operation. We have implemented HAIL on Android platforms and conducted extensive experiments in a food plaza and a premium shopping mall. Experimental results show that it achieves much higher localization accuracy (reducing the localization error by more than 20%) and computation efficiency (by more than 40% in time) as compared with the state-of-the-art approaches.

Vector line art plays an important role in graphic design, however, it is tedious to manually create. We introduce a general framework to produce line drawings from a wide variety of images, by learning a mapping from raster image space to vector image space. Our approach is based on a recurrent neural network that draws the lines one by one. A differentiable rasterization module allows for training with only supervised raster data. We use a dynamic window around a virtual pen while drawing lines, implemented with a proposed aligned cropping and differentiable pasting modules. Furthermore, we develop a stroke regularization loss that encourages the model to use fewer and longer strokes to simplify the resulting vector image. Ablation studies and comparisons with existing methods corroborate the efficiency of our approach which is able to generate visually better results in less computation time, while generalizing better to a diversity of images and applications.

No abstract