Boilerplate Removal using a Neural Sequence Labeling Model

Leonhardt, Jurek; Anand, Avishek; Khosla, Megha

doi:10.1145/3366424.3383547

Cited by 20 publications

(21 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare SemText with BoilerNet [3], Web2Text [17], and BoilerPipe [13], the best models so far, under the measures of average precision and recall. To provide fair comparison, we train and test these models as we train SemText using the same combined dataset and fine-tune them using the same development data of CleanEval and GoodTrends (except BoilerPipe that is not built for fine-tuning).…”

Section: Comparison Resultsmentioning

confidence: 99%

“…To avoid handcrafted features, deep learning models have recently been used to detect boilerplate. BoilerNet [3], for example, is a neural sequence-labeling model that represents each text block as a vector, encoding both HTML tags and words in the text block. Each index in the vector indicates the token count for a specific tag or word.…”

Section: Related Workmentioning

confidence: 99%

“…We adopt Vogels et al's method [17] to make proper alignment. The latest GoogleTrends-2017 dataset [3] published in 2020, consists of 180 documents randomly sampled from a larger pool of websites retrieved from the top Google queries with the corresponding main text. It is customary to use 80 webpages for training and the remaining 100 webpages for testing when using GoogleTrends-2017.…”

Section: A Datasetsmentioning

confidence: 99%

“…We train SemText in an end-to-end fashion on a dataset that combines three published type-1 webpages collected from 2008 to 2020, then fine-tune it using a small number of crossdomain webpages for development provided by CleanEval [2] and GoogleTrends-2017 [3]. We show that SemText achieves the highest F1 score over previous methods on these datasets.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Boilerplate Detection via Semantic Classification of TextBlocks

Zhang

Wang

2021

2021 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

We present a hierarchical neural network model called SemText to detect HTML boilerplate based on a novel semantic representation of text blocks. We train SemText on three published datasets of news webpages and fine-tune it using a small number of development data in CleanEval and GoogleTrends-2017. We show that SemText achieves the stateof-the-art accuracy on these datasets. We then demonstrate the robustness of SemText by showing that it also detects boilerplate effectively on out-of-domain community-based Q&A webpages.

show abstract

Section: Comparison Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: A Datasetsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Boilerplate Detection via Semantic Classification of TextBlocks

Zhang

Wang

2021

2021 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

show abstract

“…There are a number of other tasks that are related to semi-structured information extraction. Boilerplate removal (Leonhardt et al, 2020) attempts to remove the unrelated elements of the page, e.g. advertising or navigation, with a binary classifier.…”

Section: Related Workmentioning

confidence: 99%

PLAtE: A Large-scale Dataset for List Page Web Extraction

San¹,

Bakus²,

Lockard³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, neural models have been leveraged to significantly improve the performance of information extraction from semi-structured websites. However, a barrier for continued progress is the small number of datasets large enough to train these models. In this work, we introduce the PLAtE (Pages of Lists Attribute Extraction) dataset as a challenging new web extraction task. PLAtE focuses on shopping data, specifically extractions from product review pages with multiple items. PLAtE encompasses both the tasks of: (1) finding product-list segmentation boundaries and (2) extracting attributes for each product. PLAtE is composed of 53, 905 items from 6, 810 pages, making it the first large-scale list page web extraction dataset. We construct PLAtE by collecting list pages from Common Crawl, then annotating them on Mechanical Turk. Quantitative and qualitative analyses are performed to demonstrate PLAtE has high-quality annotations. We establish strong baseline performance on PLAtE with a SOTA model achieving an F1-score of 0.750 for attribute classification and 0.915 for segmentation, indicating opportunities for future research innovations in web extraction.

show abstract

Automatic Web Data API Creation via Cross-Lingual Neural Pagination Recognition

Chang

Lin

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Boilerplate Removal using a Neural Sequence Labeling Model

Cited by 20 publications

References 13 publications

Boilerplate Detection via Semantic Classification of TextBlocks

Boilerplate Detection via Semantic Classification of TextBlocks

PLAtE: A Large-scale Dataset for List Page Web Extraction

Automatic Web Data API Creation via Cross-Lingual Neural Pagination Recognition

Contact Info

Product

Resources

About