Image Captioning with Internal and External Knowledge

Abstract

Automatically generating a human-like description for a given image is a potential research in artificial intelligence, which has attracted a great of attention recently. Most of the existing attention methods explore the mapping relationships between words in sentence and regions in image, such unpredictable matching manner sometimes causes inharmonious alignments that may reduce the quality of generated captions. In this paper, we make our efforts to reason about more accurate and meaningful captions. We first propose word attention to improve the correctness of visual attention when generating sequential descriptions word-byword. The special word attention emphasizes on word importance when focusing on different regions of the input image, and makes full use of the internal annotation knowledge to assist the calculation of visual attention. Then, in order to reveal those incomprehensible intentions that cannot be expressed straightforwardly by machines, we inject external knowledge extracted from knowledge graph into the encoder-decoder framework to facilitate meaningful captioning. We validate our model on two freely available captioning benchmarks: Microsoft COCO dataset and Flickr30k dataset. The results demonstrate that our approach achieves state-of-the-art performance and outperforms many of the existing approaches. CCS CONCEPTS • Computing methodologies → Machine learning; Scene understanding; Natural language generation.

Semantic Web technologies and bias in artificial intelligence: A systematic literature review

“…Therefore, we investigate the combination of SW technologies with bias mitigation methods. Bias mitigation has generally been divided into three groups [56,62]: those focusing on changing the training data [2,6,19,21,24,39,41,46,47,57,85], the learning algorithm during the model generation [3,31,51,54,76,88], or the model outcomes according to the results in a holdout dataset which was not involved during the training phase [29]. Such methods may mitigate undesirable associations of specific demographic groups with hateful connotations.…”

“…From a statistical point of view, systematic deviations of the, possibly unknown, real distributions of the variables represented in the data can lead to inaccurate estimations and constitute a statistical bias. For example, representation disparities in the data of the users [29], items [2,20,65], or their recorded interactions [85] can compromise the Categories of bias depending on the location in the AI workflow where bias originates [64] Bias location Due to Reference Bias at source External bias [5,6,18,61,67] Functional bias [2,27,29,54,65,74,85] Bias at collection Sampling [34,43,47] Querying [19,76] Data pre-processing Annotation [3,14,17,20,21,24,31,38,39,41,46,51,57] Aggregation [88] Data analysis Inference and prediction [22,53,80] quality and fairness of RS. Searching for information based only on the distributions of a specific dataset can lead to irrelevant results or results biased to other meanings of the words used in the query [19,76].…”

“…Searching for information based only on the distributions of a specific dataset can lead to irrelevant results or results biased to other meanings of the words used in the query [19,76]. Similarly, the use of small, domain-specific datasets for training black-box models can lead to undesired behaviours, such as missing the image objects needed to provide meaningful captions [39] and answers to a question about the image [21], or retrieve relevant results to a search query [41], or missing the correct words that enable robots to understand the commands given in a sentence [57]. Consequently, predictions based on these datasets may lead to making decisions based on correlations that are unacceptable in specific cases, as these data only provide estimates from limited settings (e.g.…”

“…Data pre-processing is susceptible to bias, in particular in this study, of annotation and aggregation. Noisy labels due to poor or missing guidelines compromise manual annotations (of meteorological analytical data [31], a product review [46], the meaning [17] or link between words in a text [3], or the description of abnormalities in medical images [51]), and frequently lead to the use of small corpora which cannot generalise to novel examples [14,20,21,39,41,57]. In domains or problems where the ground truth may not be well defined (e.g., making a medical diagnosis), the use of annotated corpora has limited capacity to ensure that human experts reach a specific level of understanding so that these systems can be applied effectively, efficiently and satisfactorily [38].…”

“…Bias assessment Amazon KG [29] Cellosaurus, DrugBank [27] YAGO [5] SentiWordNet [18] prototype [67] FrameNet [57] Wikidata [14] prototype [38] WordNet [17] proprietary [31] SentiWordNet [46] medical KG [51] Bias representation Wikidata [61] prototype [47] MVSO [43] IMAGACT [34] DBpedia [20] TIACRITIS [80] CBOntology [53] CODM [22] Bias mitigation WordNet [6] Wikidata [54] prototype [74] Wikidata [3] DBpedia [2,65] Freebase [85] ConceptNet [88] prototype [24] ConceptNet [21,39,41] DBpedia, WebChild [21] ConceptNet, WordNet [19] prototype [76] in the metadata and normalisation to map every instance to all its possible names. This analysis generated a new resource with unduplicated and normalised data that allows examination across study platforms.…”