Sentence Extraction Based on Sentence Distribution and Part of Speech Tagging for Multi-Document Summarization

Abstract

Automatic multi-document summarization needs to find representative sentences not only b y sentence distrib ution to select the most important sentence b ut also b y how informative a term is in a sentence. Sentence distrib ution is suitab le for ob taining important sentences b y determining frequent and well-spread words in the corpus b ut ignores the grammatical information that indicates instructive content. The presence or ab sence of informative content in a sentence can b e indicated b y grammatical information which is carried b y part of speech (POS) lab els. In this paper, we propose a new sentence weighting method b y incorporating sentence distrib ution and POS tagging for multi -document summarization. Similarity-based Histogram Clustering (SHC) is used to cluster sentences in the data set. Cluster ordering is b ased on cluster importance to determine the important clusters. Sentence extraction b ased on sentence distrib ution and POS tagging is introduced to extract the representative sentences from the ordered clusters. The results of the experiment on the Document Unde rstanding Conferences (DUC) 2004 are compared with those of the Sentence Distrib ution Method. Our proposed method achieved b etter results with an increasing rate of 5.41% on ROUGE-1 and 0.62% on ROUGE-2.

Feature-based POS tagging and sentence relevance for news multi-document summarization in Bahasa Indonesia

“…In this study, the research technique from [6], [11] was adopted, and the framework by [6], [13], was also employed. The three most essential stages are conducted to obtain final summaries: i) text preprocessing, ii) sentence extraction for extracting the representative sentence, and iii) arrangement of summaries from representative sentences.…”

“…Based on Abdullah's research [6], each sentence contains grammatical information that may indicate whether or not the sentence is important to a document. Grammatical information carried by part of speech (POS) indicates to an extent the presence or absence of informative content in a sentence and increases the quality of translation [13], [17]. Part of Speech tagging is assigning a relevant tag to each term based on qualified data and even on surrounding words [18].…”

“…POS tag can mark a term that has appeared more often in the document and can be the most important term [20]. Based on Jespersen's Rank Theory, POS can be ranked into four degrees: i) nouns, because they have the most content-bearing labels, ii) adjectives, verbs, and participles, ii) adverbs, and finally iv) all remaining POS [13]. In this study, we make a list of POS label weight PW based on Jespersen's Rank Theory [21] with four weight values, and our experiment determines the PW values which can be defined as (8).…”

“…The word's expansion in the title is done to get an alternative word with the same meaning possessed by the sentence in the document. Query expansion similarity(W7) can be obtained by (13).…”

“…It concludes a compelling mixture of weighting features, i.e., word frequency, TF-IDF, lexical similarity, and sentence length [11]. Another method makes use of a statistical decision by calculating the weight of sentences based on the part of speech tagging (POS tagging) [6], [7], [12], [13]. According to the POS tagging approach, the essential sentences that turn out to be candidates for summary sentences are obtained based on the distribution of important sentences with many frequencies, spread, and most content-bearing words.…”