On document splitting in passage detection

Goharian, Nazli; Mengle, Saket S. R.

doi:10.1145/1390334.1390528

Cited by 4 publications

(4 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We apply three document splitting approaches, namely KDP, NWP, and OWP as explained in the methodology section. Prior work (Goharian & Mengle, 2008; Mengle & Goharian, 2009b) showed that KDP outperforms the other two methods in detecting passages.…”

Section: Results and Analysismentioning

confidence: 99%

“…There is no shared area between two adjacent windows; hence, these windows are called nonoverlapping windows (Hearst, 1994). In the overlapping window passage (OWP; Callan, 1994) approach, a document is divided into n ‐word passages; the overlapping windows are defined from n /2 terms of the prior passage to n /2 terms of the next passage. In the keyword‐based dynamic passage approach (KDP; Goharian & Mengle, 2008), passages are defined around the high weight terms. The probability of detecting the correct category of a passage is higher when the passage contains at least one term with a high term weight. Step 3: Classifying passages and generating confusion matrix.…”

Section: Methodsmentioning

confidence: 99%

“…In the keyword‐based dynamic passage approach (KDP; Goharian & Mengle, 2008), passages are defined around the high weight terms. The probability of detecting the correct category of a passage is higher when the passage contains at least one term with a high term weight.…”

Section: Methodsmentioning

confidence: 99%

See 2 more Smart Citations

Detecting relationships among categories using text classification

Mengle

Goharian

2010

J. Am. Soc. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

Discovering relationships among concepts and categories is crucial in various information systems. The authors' objective was to discover such relationships among document categories. Traditionally, such relationships are represented in the form of a concept hierarchy, grouping some categories under the same parent category. Although the nature of hierarchy supports the identification of categories that may share the same parent, not all of these categories have a relationship with each other-other than sharing the same parent. However, some "non-sibling"relationships exist that although are related to each other are not identified as such. The authors identify and build a relationship network (relationship-net) with categories as the vertices and relationships as the edges of this network. They demonstrate that using a relationship-net, some nonobvious category relationships are detected. Their approach capitalizes on the misclassification information generated during the process of text classification to identify potential relationships among categories and automatically generate relationship-nets. Their results demonstrate a statistically significant improvement over the current approach by up to 73% on 20 News groups 20NG, up to 68% on 17 categories in the Open Directories Project (ODP17), and more than twice on ODP46 and Special Interest Group on Information Retrieval (SIGIR) data sets. Their results also indicate that using misclassification information stemming from passage classification as opposed to document classification statistically significantly improves the results on 20NG (8%), ODP17 (5%), ODP46 (73%), and SIGIR (117%) with respect to F1 measure. By assigning weights to relationships and by performing feature selection, results are further optimized.

show abstract

Section: Results and Analysismentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Detecting relationships among categories using text classification

Mengle

Goharian

2010

J. Am. Soc. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the keyword‐based dynamic passage (KDP) approach (Goharian & Mengle, 2008), passages are defined around terms with higher weights. We assume that the probability of detecting the correct category of a passage is higher when the passage contains a term with a higher weight.…”

Section: Methodsmentioning

confidence: 99%

Passage detection using text classification

Mengle¹,

Goharian²

2009

J. Am. Soc. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

Passages can be hidden within a text to circumvent their disallowed transfer. Such release of compartmentalized information is of concern to all corporate and governmental organizations. Passage retrieval is well studied; we posit, however, that passage detection is not. Passage retrieval is the determination of the degree of relevance of blocks of text, namely passages, comprising a document. Rather than determining the relevance of a document in its entirety, passage retrieval determines the relevance of the individual passages. As such, modified traditional information-retrieval techniques compare terms found in user queries with the individual passages to determine a similarity score for passages of interest. In passage detection, passages are classified into predetermined categories. More often than not, passage detection techniques are deployed to detect hidden paragraphs in documents. That is, to hide information, documents are injected with hidden text into passages. Rather than matching query terms against passages to determine their relevance, using text-mining techniques, the passages are classified. Those documents with hidden passages are defined as infected. Thus, simply stated, passage retrieval is the search for passages relevant to a user query, while passage detection is the classification of passages. That is, in passage detection, passages are labeled with one or more categories from a set of predetermined categories. We present a keyword-based dynamic passage approach (KDP) and demonstrate that KDP outperforms statistically significantly (99% confidence) the other document-splitting approaches by 12% to 18% in the passage detection and passage category-prediction tasks. Furthermore, we evaluate the effects of the feature selection, passage length, ambiguous passages, and finally training-data category distribution on passage-detection accuracy.

show abstract