A Synchronized Word Representation Method With Dual Perceptual Information

Abstract

The information used for human natural language comprehension is usually perceptual information, such as text, sounds, and images. In recent years, language models that learn semantics from single perceptual information sources (text) have gradually developed into multimodal language models that learn semantics from multiple perceptual information sources. Sound is perceptual information other than text that has been proven effective by many related works. However, there is still a need for further research on the incorporation method for perceptual information. Thus, this paper proposes a language model that synchronously trains dual perceptual information to enhance word representation. The representation is trained in a synchronized way that adopts an attention model to utilize both text and phonetic perceptual information in unsupervised learning tasks. On basis of that, these dual perceptual information is processed simultaneously, and that is similar with the cognitive process of human language understanding. The experiment results show that our approach achieve superior results in text classification and word similarity tasks with four languages of data set. INDEX TERMS Information representation, multi-layer neural network, natural language processing, unsupervised learning.

Multimodal Sentiment Analysis of Online Product Information Based on Text Mining Under the Influence of Social Media

“…The distributed representation method, also known as word embedding, inputs the words in the text into a pre-trained model for training, and then converts it into a continuous dense vector. This method uses the distribution assumption of words to represent texts, that is, words with similar contexts are described as similar semantics so that words with similar contexts are close in the semantic space (Zhu et al, 2020). The Word2Vec and Paragraph Vector models in the word distribution representation are introduced below, as shown in Figure 1.…”

Learning Multimodal Word Representations by Explicitly Embedding Syntactic and Phonetic Information

“…It trained written embedding based on phonetic embedding and the final word representation fuses writing and phonetic embedding. W. Zhu et al [63] use a synchronized way that adopts an attention model to utilize both text and phonetic perceptual information in unsupervised learning tasks. In terms of the two types of models discussed in this section, MSP belongs to separate training model.…”

“…DCWE [58] is enhanced double-carrier word representation model via phonetics and writing, and it trained written representation based on phonetic representation and the final word representation fuses text and phonetic embedding. DPWR [63] is trained in a synchronized way that adopts an attention model to utilize both linguistic and phonetic information in unsupervised learning tasks. SynGCN [57] incorporates syntactic and semantic information in word embeddings by using graph convolutional networks.…”