Corpus-based web document summarization using statistical and linguistic approach

Abstract

Single document summarization generates summary by extracting the representative sentences from the document. In this paper, we presented a novel technique for summarization of domain-specific text from a single web document that uses statistical and linguistic analysis on the text in a reference corpus and the web document. The proposed summarizer uses the combinational function of Sentence Weight ( ) and Subject Weight ( ) to determine the rank of a sentence, where is the function of number of terms ( ) and number of words ( ) in a sentence, and term frequency ( ) in the corpus and is the function of and in a subject, and in the corpus. 30 percent of the ranked sentences are considered to be the summary of the web document. We generated three web document summaries using our technique and compared each of them with the summaries developed manually from 16 different human subjects. Results showed that 68 percent of the summaries produced by our approach satisfy the manual summaries.

Domain-specific textual commonsense concept acquisition using a corpus

“…Suanmali et al proposed feature based sentence and knowledge extraction technique in [16], where representativeness of sentences and knowledge depended on fused features like normalization, term frequencies and number of proper nouns. We did not fuse these features rather we used them independently as [9] showed that independent features perform better on domain-specific information retrieval. Cao et al [2] weighed adjectives to extract concepts from commonsense knowledge.…”

“…For example, a list of Term Frequencies [9] is given in Table II. When considering domain-specific knowledge, it is usual that some commonsense will be present in more than one knowledge.…”

“…This denotes the statistical distance of a knowledge from others due to the variation of commonsense present in them. Moreover, if is the sentence whose textual commonsense concepts are to be acquired, we calculated the sentence weight , which is the function of number of terms ( ), number of words ( ) and term frequencies ( ) and normalized it to get [9]. Then, we selected the relevant commonsense knowledge using statistical analysis on and and selected the proper nouns in them.…”

Artificial Intelligence in Business Communication: A Snapshot

“…Since then, a large number of techniques and approaches have been developed. Interestingly, the large volumes of information created on the web have triggered much of this development (Nenkova & McKeown, 2011;Shams, Hashem, Hossain, Akter, & Gope, 2010). Bhargava, Sharma, and Sharma (2016) posit that text summarization tools have now become a necessity to navigate the information on the web because they help eliminate dispensable or superfluous content.…”

“…Text units with a concentration of high-score words are often likely contenders for extraction (Liu & Liu, 2009). Extraction-based summarization, then, is essentially concerned with evaluating the salience or the indicative power of each sentence in a given document (Shams et al, 2010). Figure 1 maps out the process flow for extraction-based systems.…”