Previous studies in multimodal sentiment analysis have used limited datasets, which only contain unified multimodal annotations. However, the unified annotations do not always reflect the independent sentiment of single modalities and limit the model to capture the difference between modalities. In this paper, we introduce a Chinese single-and multimodal sentiment analysis dataset, CH-SIMS, which contains 2,281 refined video segments in the wild with both multimodal and independent unimodal annotations. It allows researchers to study the interaction between modalities or use independent unimodal annotations for unimodal sentiment analysis. Furthermore, we propose a multi-task learning framework based on late fusion as the baseline. Extensive experiments on the CH-SIMS show that our methods achieve state-of-the-art performance and learn more distinctive unimodal representations. The full dataset and codes are available for use at https://github.com/ thuiar/MMSA.

Image-text matching task aims to learn the fine-grained correspondences between images and sentences. Existing methods use attention mechanism to learn the correspondences by attending to all fragments without considering the relationship between fragments and global semantics, which inevitably lead to semantic misalignment among irrelevant fragments. To this end, we propose a Bidirectional Correct Attention Network (BCAN), which leverages global similarities and local similarities to reassign the attention weight, to avoid such semantic misalignment. Specifically, we introduce a global correct unit to correct the attention focused on relevant fragments in irrelevant semantics. A local correct unit is used to correct the attention focused on irrelevant fragments in relevant semantics. Experiments on Flickr30K and MSCOCO datasets verify the effectiveness of our proposed BCAN by outperforming both previous attention-based methods and state-of-the-art methods. Code can be found at: https://github.com/liuyyy111/BCAN.

To predict the epidermal growth factor receptor (EGFR) mutation status in patients with lung adenocarcinoma using quantitative radiomic biomarkers and semantic features. We analyzed the computed tomography (CT) images and medical record data of 104 patients with lung adenocarcinoma who underwent surgical excision and EGFR mutation detection from 2016 to 2018 at our center. CT radiomic and semantic features that reflect the tumors’ heterogeneity and phenotype were extracted from preoperative non-enhanced CT scans. The least absolute shrinkage and selection operator method was applied to select the most distinguishable features. Three logistic regression models were built to predict the EGFR mutation status by combining the CT semantic with clinicopathological characteristics, using the radiomic features alone, and by combining the radiomic and clinicopathological features. Receiver operating characteristic (ROC) curve analysis was performed using five-fold cross-validation and the mean area under the curve (AUC) values were calculated and compared between the models to obtain the optimal model for predicting EGFR mutation. Furthermore, radiomic nomograms were constructed to demonstrate the performance of the model. In total, 1025 radiomic features were extracted and reduced to 13 features as the most important predictors to build the radiomic signature. The combined radiomic and clinicopathological features model was developed based on the radiomic signature, sex, smoking, vascular infiltration, and pathohistological type. The AUC was 0.90  ±  0.02 for the training, 0.88  ±  0.11 for the verification, and 0.894 for the test dataset. This model was superior to the other prediction models that used the combined CT semantic and clinicopathological features (AUC for the test dataset: 0.768) and radiomic features alone (AUC for the test dataset: 0.837). The prediction model built by radiomic biomarkers and clinicopathological features, including the radiomic signature, sex, smoking, vascular infiltration, and pathological type, outperformed the other two models and could effectively predict the EGFR mutation status in patients with peripheral lung adenocarcinoma. The radiomic nomogram of this model is expected to become an effective biomarker for patients with lung adenocarcinoma requiring adjuvant targeted treatment.

Recently, unsupervised methods for monocular visual odometry (VO), with no need for quantities of expensive labeled ground truth, have attracted much attention. However, these methods are inadequate for long-term odometry task, due to the inherent limitation of only using monocular visual data and the inability to handle the error accumulation problem. By utilizing supplemental low-cost inertial measurements, and exploiting the multi-view geometric constraint and sequential constraint, an unsupervised visual-inertial odometry framework (UnVIO) is proposed in this paper. Our method is able to predict the per-frame depth map, as well as extracting and self-adaptively fusing visual-inertial motion features from image-IMU stream to achieve long-term odometry task. A novel sliding window optimization strategy, which consists of an intra-window and an inter-window optimization, is introduced for overcoming the error accumulation and scale ambiguity problem. The intra-window optimization restrains the geometric inferences within the window through checking the photometric consistency. And the inter-window optimization checks the 3D geometric consistency and trajectory consistency among predictions of separate windows. Extensive experiments have been conducted on KITTI and Malaga datasets to demonstrate the superiority of UnVIO over other state-of-the-art VO / VIO methods. The codes are open-source.

Highlights It is vital to distinguish indolent pulmonary adenocarcinomas from invasive pulmonary adenocarcinomas before surgery. Radiomics is a cutting-edge technology that mines quantitative features from CT images. We designed a nomogram, which incorporated clinical and CT morphological characteristics with the radiomics signature. We applied the radiomics nomogram to preoperatively predict the invasiveness of GGNs.