Automatically generating a natural language description of an image has attracted interests recently both because of its importance in practical applications and because it connects two major artificial intelligence fields: computer vision and natural language processing. Existing approaches are either top-down, which start from a gist of an image and convert it into words, or bottom-up, which come up with words describing various aspects of an image and then combine them. In this paper, we propose a new algorithm that combines both approaches through a model of semantic attention. Our algorithm learns to selectively attend to semantic concept proposals and fuse them into hidden states and outputs of recurrent neural networks. The selection and fusion form a feedback connecting the top-down and bottom-up computation. We evaluate our algorithm on two public benchmarks: Microsoft COCO and Flickr30K. Experimental results show that our algorithm significantly outperforms the state-of-the-art approaches consistently across different evaluation metrics.

Deep learning techniques have been successfully applied in many areas of computer vision, including low-level image restoration problems. For image super-resolution, several models based on deep neural networks have been recently proposed and attained superior performance that overshadows all previous handcrafted models. The question then arises whether large-capacity and data-driven models have become the dominant solution to the ill-posed superresolution problem. In this paper, we argue that domain expertise represented by the conventional sparse coding model is still valuable, and it can be combined with the key ingredients of deep learning to achieve further improved results. We show that a sparse coding model particularly designed for super-resolution can be incarnated as a neural network, and trained in a cascaded structure from end to end. The interpretation of the network based on sparse coding leads to much more efficient and effective training, as well as a reduced model size. Our model is evaluated on a wide range of images, and shows clear advantage over existing state-of-the-art methods in terms of both restoration accuracy and human subjective quality.

Due to the significant information loss in low-resolution (LR) images, it has become extremely challenging to further advance the state-of-the-art of single image super-resolution (SISR). Reference-based super-resolution (RefSR), on the other hand, has proven to be promising in recovering high-resolution (HR) details when a reference (Ref) image with similar content as that of the LR input is given. However, the quality of RefSR can degrade severely when Ref is less similar. This paper aims to unleash the potential of RefSR by leveraging more texture details from Ref images with stronger robustness even when irrelevant Ref images are provided. Inspired by the recent work on image stylization, we formulate the RefSR problem as neural texture transfer. We design an end-to-end deep model which enriches HR details by adaptively transferring the texture from Ref images according to their textural similarity. Instead of matching content in the raw pixel space as done by previous methods, our key contribution is a multi-level matching conducted in the neural space. This matching scheme facilitates multi-scale neural transfer that allows the model to benefit more from those semantically related Ref patches, and gracefully degrade to SISR performance on the least relevant Ref inputs. We build a benchmark dataset for the general research of RefSR, which contains Ref images paired with LR inputs with varying levels of similarity. Both quantitative and qualitative evaluations demonstrate the superiority of our method over state-of-the-art 1 .

Recent progresses on deep discriminative and generative modeling have shown promising results on texture synthesis. However, existing feed-forward based methods trade off generality for efficiency, which suffer from many issues, such as shortage of generality (i.e., build one network per texture), lack of diversity (i.e., always produce visually identical output) and suboptimality (i.e., generate less satisfying visual effects). In this work, we focus on solving these issues for improved texture synthesis. We propose a deep generative feed-forward network which enables efficient synthesis of multiple textures within one single network and meaningful interpolation between them. Meanwhile, a suite of important techniques are introduced to achieve better convergence and diversity. With extensive experiments, we demonstrate the effectiveness of the proposed model and techniques for synthesizing a large number of textures and show its applications with the stylization.

Rab small GTPases are involved in the transport of vesicles between different membranous organelles. RAB-3 is an exocytic Rab that plays a modulatory role in synaptic transmission. Unexpectedly, mutations in the Caenorhabditis elegans RAB-3 exchange factor homologue, aex-3, cause a more severe synaptic transmission defect as well as a defecation defect not seen in rab-3 mutants. We hypothesized that AEX-3 may regulate a second Rab that regulates these processes with RAB-3. We found that AEX-3 regulates another exocytic Rab, RAB-27. Here, we show that C. elegans RAB-27 is localized to synapse-rich regions pan-neuronally and is also expressed in intestinal cells. We identify aex-6 alleles as containing mutations in rab-27. Interestingly, aex-6 mutants exhibit the same defecation defect as aex-3 mutants. aex-6; rab-3 double mutants have behavioral and pharmacological defects similar to aex-3 mutants. In addition, we demonstrate that RBF-1 (rabphilin) is an effector of RAB-27. Therefore, our work demonstrates that AEX-3 regulates both RAB-3 and RAB-27, that both RAB-3 and RAB-27 regulate synaptic transmission, and that RAB-27 potentially acts through its effector RBF-1 to promote soluble N-ethylmaleimide-sensitive factor attachment protein receptor (SNARE) function. INTRODUCTIONNeurotransmitter release is accomplished by the fusion of neurotransmitter-filled synaptic vesicles at the presynaptic nerve terminal. This process occurs through a series of highly regulated steps that include synaptic vesicle transport, docking/tethering, priming, fusion, endocytosis, recycling, and neurotransmitter refilling (Sudhof, 2004). Several genes have been assigned roles in the various steps of the synaptic vesicle cycle. Rab3 (termed RAB-3 in Caenorhabditis elegans), a member of the Rab family of small GTPases, regulates synaptic transmission, possibly through the docking, priming, or fusion steps Schluter et al., 2004).Rabs act in a variety of cell types and regulate vesicular transport between organelles. Rabs cycle on and off membranes via a GTP-dependent mechanism (Zerial and McBride, 2001). Rab activity is regulated by two proteins, which act in an antagonistic manner. The guanine nucleotide exchange factor (GEF) exchanges GTP for GDP, and the GTPase activating protein activates the intrinsic GTPase activity of a Rab (Bernards, 2003). GDP bound Rabs are held off of the membrane by a GDP dissociation inhibitor (Wu et al., 1996). The GTP/membrane-bound form of Rabs typically binds to a variety of effectors that regulate particular steps of membrane transport and cell signaling (Zerial and McBride, 2001;Spang, 2004).Rab3 was once thought to play a central role in regulating release. However, recent work has shown that a quadruple knockout of all four isoforms of Rab3 in mice only results in a 30% reduction in evoked synaptic response (Schluter et al., 2004). This is consistent with work in C. elegans showing that mutations in the single rab-3 gene cause only mild defects in synaptic transmission . Surprisingly, more dramatic phe...