Image captioning aims to automatically generate a natural language description of a given image, and most state-ofthe-art models have adopted an encoder-decoder framework. The framework consists of a convolution neural network (CNN)-based image encoder that extracts region-based visual features from the input image, and an recurrent neural network (RNN)-based caption decoder that generates the output caption words based on the visual features with the attention mechanism. Despite the success of existing studies, current methods only model the co-attention that characterizes the inter-modal interactions while neglecting the self-attention that characterizes the intramodal interactions. Inspired by the success of the Transformer model in machine translation, here we extend it to a Multimodal Transformer (MT) model for image captioning. Compared to existing image captioning approaches, the MT model simultaneously captures intra-and inter-modal interactions in a unified attention block. Due to the in-depth modular composition of such attention blocks, the MT model can perform complex multimodal reasoning and output accurate captions. Moreover, to further improve the image captioning performance, multi-view visual features are seamlessly introduced into the MT model. We quantitatively and qualitatively evaluate our approach using the benchmark MSCOCO image captioning dataset and conduct extensive ablation studies to investigate the reasons behind its effectiveness. The experimental results show that our method significantly outperforms the previous state-of-the-art methods. With an ensemble of seven models, our solution ranks the 1st place on the real-time leaderboard of the MSCOCO image captioning challenge at the time of the writing of this paper.

A key issue in drug development is to understand the hidden relationships among drugs and targets. Computational methods for novel drug target predictions can greatly reduce time and costs compared with experimental methods. In this paper, we propose a network based computational approach for novel drug and target association predictions. More specifically, a heterogeneous drug-target graph, which incorporates known drug-target interactions as well as drug-drug and target-target similarities, is first constructed. Based on this graph, a novel graph-based inference method is introduced. Compared with two state-of-the-art methods, large-scale cross-validation results indicate that the proposed method can greatly improve novel target predictions.

No abstract

Aim of this study was to develop a time-efficient sequence protocol for a 1.0 T dedicated MR system to be used for whole-organ scoring of osteoarthritis (OA). Thirty-four knees were examined using a protocol that included fat suppressed fast spin echo proton density weighted sequences (PDFS) in three planes plus a coronal STIR sequence. Two radiologists scored each knee by consensus for five OA features. In separate sessions, all knees were scored using three different combinations of sequences: (1) all four sequences (reference protocol, 16 min 31 s scanning time), (2) three PDFS sequences without STIR ("No STIR", 12 min 25 s scanning time) and (3) sagittal and axial PDFS sequences plus a coronal STIR sequence ("No PDFS", 11 min 49 s scanning time). Agreement of the readings using both subsets of sequences compared to the reference protocol was evaluated using weighted kappa statistics. kappa-coefficients showed good or excellent agreement for both sequence subsets in comparison to the reference protocol for all assessed features. kappa-coefficients for No PDFS/No STIR: bone marrow abnormalities (0.74/0.67), subarticular cysts (0.84/0.63), marginal osteophytes (0.77/0.71), menisci (0.75/0.79), tibial cartilage (0.71/0.78). Optimization of sequence protocols consisting of three sequences results in time savings and cost efficiency in imaging of knee OA without loss of information over a more time consuming protocol.

No abstract