Automatic text analysis based on transition phenomena of word occurrences

Abstract

A method of selecting index terms directly from a word frequency list is described. The original idea was suggested by Goffman who reasoned that the most content-bearing words of a given text would occur at the transition region at which Zipf's First Law of words of high frequency of occurrences begins to take on properties of words of low frequency of occurrences. Word frequencies of two articles were analyzed. Results seem to indicate that the automated selection of index terms from a frequency list holds some promise for automatic indexing.

Gestão em Unidades de Terapia Intensiva Brasileiras: Estudo Bibliométrico dos Últimos 10 Anos

“…A Lei de Bradford está relacionada à dispersão da literatura científica publicada em periódicos, isto é, no início um tema é publicado em alguns periódicos e, posteriormente atinge um número progressivamente maior de revistas sobre determinado assunto, podendo ser dividido em ordem decrescente de produtividade classificados em zonas, sendo a zona com mais publicações chamada de zona nuclear (Brookes, 1969). Por fim, a Lei de Zipf que se baseia na frequência da ocorrência de palavras ao analisar um texto longo, concluindo que um pequeno número de palavras é usado muito mais frequentemente, podendo-se listá-las em sistema de ranqueamento (Pao, 1978).…”

Document Clustering with Evolutionary Systems through Straight-Line Programs “<i>slp</i>”

“…In this model documents are represented by vector of terms that they are obtained with NZIPF method [7]. We use term vectors, obtained applying a methodology of processing the documents [8] to obtain the best characteristic vectors, selected from the area defined by the use of the Zipf law [9] and the Goffman Point [10,11].…”

A Luhn-Inspired Vector Re-weighting Approach for Improving Personalized Web Search

“…We discuss the issues of choosing these two parameters separately. 1) Determining the Location: In order to choose the mean value for the normal distribution used in the vector re-weighting process, we find that Zipf's Laws, and more importantly, Goffman's theory regarding the transition region [15], provides valuable information for our purposes. Consistent with Luhn's model, Goffman pointed out that there is a transition region between Zipf's First Law of highfrequency words [10] and Booth's revision of Zipf's Second Law of low-frequency words [16].…”