The Role of the Input in Natural Language Video Description

Abstract

Natural Language Video Description (NLVD) has recently received strong interest in the Computer Vision, Natural Language Processing (NLP), Multimedia, and Autonomous Robotics communities. The State-of-the-Art (SotA) approaches obtained remarkable results when tested on the benchmark datasets. However, those approaches poorly generalize to new datasets. In addition, none of the existing works focus on the processing of the input to the NLVD systems, which is both visual and textual. In this work, it is presented an extensive study dealing with the role of the visual input, evaluated with respect to the overall NLP performance. This is achieved performing data augmentation of the visual component, applying common transformations to model camera distortions, noise, lighting, and camera positioning, that are typical in real-world operative scenarios. A t-SNE based analysis is proposed to evaluate the effects of the considered transformations on the overall visual data distribution. For this study, it is considered the English subset of Microsoft Research Video Description (MSVD) dataset, which is used commonly for NLVD. It was observed that this dataset contains a relevant amount of syntactic and semantic errors. These errors have been amended manually, and the new version of the dataset (called MSVD-v2) is used in the experimentation. The MSVD-v2 dataset is released to help to gain insight into the NLVD problem.

The Comparative study of Python Libraries for Natural Language Processing (NLP)

“…Due to the large amounts of data that are accessible on the Internet, Natural Language Processing (NLP) has also experienced a shift towards information extraction and production during the last decade. [29] Additionally, as personal computers have become more accessible to the general public, applications for natural language processing (NLP) have grown more widespread, which has stimulated additional study in the subject. [8] III.OVERVIEW OF NLP "Natural Language Processing (NLP) is an area of computing technology, specifically within the domain of artificial cognition (AI), that is devoted to providing computers with the capacity to grasp written text and spoken language in a manner that is comparable to how human beings do so.…”

Research on image analysis and processing method based on compressed perception technology

“…With the rapid development of the economy and society and the continuous progress of scientific and technological power, video images play a rather important role in information communication [1]. Relevant scientific data show that about 80% of human's access to external information comes from vision, which shows that vision is an important channel for humans to know and transform the world, so the effective transmission and expression of video image information is of great importance [2][3].…”