Unsupervised acquisition of idiomatic units of symbolic natural language: An n-gram frequency-based approach for the chunking of news articles and tweets

Abstract

Symbolic sequential data are produced in huge quantities in numerous contexts, such as text and speech data, biometrics, genomics, financial market indexes, music sheets, and online social media posts. In this paper, an unsupervised approach for the chunking of idiomatic units of sequential text data is presented. Text chunking refers to the task of splitting a string of textual information into non-overlapping groups of related units. This is a fundamental problem in numerous fields where understanding the relation between raw units of symbolic sequential data is relevant. Existing methods are based primarily on supervised and semi-supervised learning approaches; however, in this study, a novel unsupervised approach is proposed based on the existing concept of n-grams, which requires no labeled text as an input. The proposed methodology is applied to two natural language corpora: a Wall Street Journal corpus and a Twitter corpus. In both cases, the corpus length was increased gradually to measure the accuracy with a different number of unitary elements as inputs. Both corpora reveal improvements in accuracy proportional with increases in the number of tokens. For the Twitter corpus, the increase in accuracy follows a linear trend. The results show that the proposed methodology can achieve a higher accuracy with incremental usage. A future study will aim at designing an iterative system for the proposed methodology.

Police alarm address recognition and classification based on convolutional neural networks

“…This method is simple but can easily produce multiple useless words. The statistics-based word classification method works on the basis of n-grams (Dario et al, 2020). The method should first prepare a complete corpus for training and extract the features of the text in accordance with the number of occurrences between adjacent characters.…”

“…It is not only one of the most representative algorithms of deep learning, but also a core algorithm in the image recognition field. It is widely used in computer vision, natural language processing, remote sensing, atmospheric science, and other fields (Dario et al, 2020).…”

Content-based user classifier to uncover information exchange in disaster-motivated networks

“…The current literature shows the importance of disseminating information during a disaster [ 1 , 2 , 7 , 8 , 19 , 20 ], but there is still research to be done in terms of the speed, accuracy and diffusion rates that can be measured during a disaster. This work addresses the following research questions: Which electronic media based information is useful and accurate during a disaster and how do we identify it?…”