Comparative Study Of Various Scraping Tools: Pros And Cons

Matta, Priya; Sharma, Sonal; Uniyal, Nitin

doi:10.1109/delcon54057.2022.9753358

Cited by 4 publications

(1 citation statement)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this sense, the web-scraping technologies applied, i.e., Selenium and XML parsing, were favorable for our data-extraction objectives and are recommended in the literature [56][57][58], but other open-source alternatives, such as BeautifulSoup, Scrapy, or Puppeteer for the Python [59] and Node.js [60] languages, are not less important. In addition, there are also commercial software alternatives which simplify web scraping when users are not familiar with programming such as Octoparse, ParseHub, or WebHarvy [61]. However, some particularities should be mentioned.…”

Section: Possibilities and Challenges Of Web Scraping Data Imputation...mentioning

confidence: 99%

An Approach Based on Web Scraping and Denoising Encoders to Curate Food Security Datasets

Santos

Acosta

2023

Agriculture

View full text Add to dashboard Cite

Ensuring food security requires the publication of data in a timely manner, but often this information is not properly documented and evaluated. Therefore, the combination of databases from multiple sources is a common practice to curate the data and corroborate the results; however, this also results in incomplete cases. These tasks are often labor-intensive since they require a case-wise review to obtain the requested and completed information. To address these problems, an approach based on Selenium web-scraping software and the multiple imputation denoising autoencoders (MIDAS) algorithm is presented for a case study in Ecuador. The objective was to produce a multidimensional database, free of data gaps, with 72 species of food crops based on the data from 3 different open data web databases. This methodology resulted in an analysis-ready dataset with 43 parameters describing plant traits, nutritional composition, and planted areas of food crops, whose imputed data obtained an R-square of 0.84 for a control numerical parameter selected for validation. This enriched dataset was later clustered with K-means to report unprecedented insights into food crops cultivated in Ecuador. The methodology is useful for users who need to collect and curate data from different sources in a semi-automatic fashion.

show abstract