Comparison of Web Scraping Techniques : Regular Expression, HTML DOM and Xpath

Gunawan, Rohmat; Rahmatulloh, Alam; Darmawan, Irfan; Firdaus, Firman Shakti

doi:10.2991/icoiese-18.2019.50

Cited by 32 publications

(27 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Gunawan et al 4 compare the three different techniques for web scraping that is, Regular Expression (Regex), HTML DOM, and XPath in terms of performance. The research through experimentation, concludes by terming Regex technique as better in “less consumption of memory” than the other two, while HTML Dom technique performed better, in terms of “time.” Parvez et al 5 discusses about the various data extraction techniques and compare those, in relation to services.…”

Section: Related Workmentioning

confidence: 99%

Smart algorithmic based web crawling and scraping with template autoupdate capabilities

Khan

Tsaramirsis

Ullah³

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

Summary Web scraping is the process of extracting data from web pages and it is an essential part for the generation of datasets. Currently the field is dominated by capable commercial applications, however, there is always a need for web crawling and web scraping applications for custom projects. Developing fit for purpose tools for retrieving and structuring data from web services, cloud systems, and big data is a challenging task. Based on empirical studies, some of the challenges include structural issues, formatting/ presentation, availability, denial of service, size, and information fetching problems with browsers. Additionally, the data become inaccessible after the structure/template of the website changes for example, after the website update. Thus the dataset cannot be updated in the future without manually modifying the parameters of the Web Scraper. In this paper we propose an algorithm capable of autocorrecting the template (web scraping parameters) used for locating the target data and dealing with some common empirical problems. This is very useful in case there is a need for updating the dataset later, as usually, websites tend to change their pages. Moreover, we introduce an implementation of the algorithm via a tool developed for extracting data from the unity asset store. The tool can capture and store data in XML format. The tool extracted a total of 46 785 (40 611 3D and 6174 2D) items, with 35 successful first retries, 11 second retries and 5 fails.

show abstract

Section: Related Workmentioning

confidence: 99%

Smart algorithmic based web crawling and scraping with template autoupdate capabilities

Khan

Tsaramirsis

Ullah³

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Salah satu cara untuk mengambil data dari web adalah web sraping. Teknologi web scraping sudah banyak digunakan diantaranya pada penelitian [5], sementara berdasarkan penelitian [6] mengenai pengambilan data menggunakan web scraping dan HTML DOM, telah berhasil dilakukan pengambilan data untuk membangun korpus paralel dari data hasil scraping dengan berbagai format data secara otomatis, sehingga pada sistem LSP Universitas Siliwangi akan diterapkan teknik web scraping.…”

Section: Pendahuluanunclassified

“…Pada penelitian yang akan dilakukan untuk melengkapi kekurangan-kekurangan pada penelitian sebelumnya akan diterapkan Teknik Web Scraping yang sering dikenal sebagai screen scraping adalah teknik pengambilan sebuah dokumen semi terstruktur dalam bahasa markup seperti HTML atau XHTML, dan menganalisis dokumen tersebut untuk diambil data tertentu dan dimanfaatkan dalam berbagai kepentingan [12]. Berdasarakan penelitian sebelumnya [6] perbandingan Teknik web scraping HTML DOM memiliki penggunaan memori paling sedikit. HTML DOM adalah sebuah library untuk mendapatkan, mengubah, menambah, atau menghapus elemen HTML.…”

Section: E-issn:2540-9719unclassified

Implementasi Web Scraping Pada Sistem Informasi Lembaga Sertifikasi Profesi Universitas Siliwangi

Akbar¹,

Rahmatulloh

Daely³

2019

SISTEMASI

Self Cite

View full text Add to dashboard Cite

ABSTRAK Lembaga Sertifikasi Profesi Universitas Siliwangi yang baru di dirikan pada tahun 2017 dalam pelakasanaan pelayanan permohonan sertifikasi profesi belum secara komputerisasi, selain itu data dan informasi belum terhubung dengan website BNSP. Sehingga banyak terjadi redudansi data dan masih banyak data dan informasi yang harus diinput manual berulang. Diperlukan Teknik khusus untuk mendapatkan informasi dan data secara otomatis dari website BNSP. Salah satu cara untuk pertukaran data yaitu dengan memanfaatkan API (Application programing interface), namun belum tersedia pada website BNSP, sehingga cara lain dapat menggunakan Teknik Web Scraping dengan menggunakan HTML DOM agar data dan informasi BNSP dapat langsung diambil untuk sistem LSP Universitas Siliwangi. Hasil penelitian ini telah berhasil dibuat sistem dengan menerapkan Teknik web scraping sehingga sistem LSP UNSIL mendapatkan data dan informasi terkini secara otomatis.

show abstract

“…Several studies related to the implementation of web scraping of scientific article or literature from the internet have been carried out beforehand including: web scraping for Indonesian -English parallel corpus using HTML DOM method [4], web-scraping software in searching for gray literature [5], application of web scraping techniques in scientific article search engines [13], the application of web scraping and winnowing web for the detection of plagiarism in the final project title [14], [15]. There are several algorithms that can be used in web scraping such as: regular expressions, HTML DOM, and Xpath [16]. Each algorithm has its own characteristics, so it needs a good understanding before applying it.…”

Section: Introductionmentioning

confidence: 99%

Web Scraping with HTML DOM Method for Data Collection of Scientific Articles from Google Scholar

Rahmatulloh¹,

Gunawan²

2020

Indonesian J. of Inf. Syst.

Self Cite

View full text Add to dashboard Cite

Abstrak. Google Scholar adalah layanan berbasis web untuk mencari literatur akademik. Berbagai jenis referensi yang dapat diakses contohnya adalah: makalah peer-review, tesis, buku, abstrak dan artikel dari penerbit akademik, komunitas profesional, pusat data pra-cetak, universitas dan organisasi akademik lainnya. Google Scholar menyediakan fitur pembuatan profil untuk setiap peneliti, pakar, dan dosen. Jumlah publikasi dari lembaga akademis bersama dengan data yang detail tentang publikasi artikel ilmiah dapat diakses melalui Google Scholar. Rekapitulasi publikasi artikel ilmiah dari masing-masing peneliti di suatu lembaga diperlukan untuk menentukan kinerja penelitian secara kolektif. Namun, hal ini masih menyisakan masalah, yaitu belum tersedianya layanan rekapitulasi publikasi artikel ilmiah untuk setiap peneliti di suatu lembaga. Oleh karena itu, penelitian ini berupaya melakukan rekapitulasi publikasi artikel ilmiah. Pengumpulan data dari Google Scholar dilakukan dengan menerapkan teknologi web scrapping. Eksperimen web scrapping dari Google Scholar dalam penelitian ini telah berhasil mengambil 238 data peneliti dan 2.523 file artikel. Data yang telah diunduh disimpan ke dalam basis data, kemudian digunakan untuk rekapitulasi publikasi artikel ilmiah, yang dapat menampilkan: daftar profil peneliti, daftar afiliasi, daftar kutipan, dan daftar judul artikel yang dapat dicetak dalam bentuk *.pdf atau *.xlsx dan dilengkapi dengan pencarian data dan fitur penyortiran.Abstract. Google Scholar is a web-based service for searching broad academic literature. Various types of references can be accessed such as: peer-reviewed papers, theses, books, abstracts and articles from academic publishers, professional communities, pre-printed data center, universities and other academic organizations. Google Scholar provides the profile creation feature of every researcher, expert, and lecturer. The quantity of publication from an academic institution along with detailed data on the publication of scientific articles can be accessed through Google Scholar. Recapitulation of the publication of scientific articles of each researcher in an institution is needed to determine the research performance collectively. However, it still leaves a problem, that is the unavailability of recapitulation services publication of scientific articles for each researcher in an institution. Therefore, this study attempts to make the recapitulation of scientific article publications. Data collection from Google Scholar was carried out by applying web scraping technology. The scraping experiment from Google Scholar in this study has succeeded in retrieving 238 researchers' data and 2,523 article files. The data that had been downloaded was stored in a database, then used to recapitulate the publication of scientific articles, which can display: a list of researcher profiles, a list of affiliations, a list of

show abstract

Comparison of Web Scraping Techniques : Regular Expression, HTML DOM and Xpath

Cited by 32 publications

References 9 publications

Smart algorithmic based web crawling and scraping with template autoupdate capabilities

Smart algorithmic based web crawling and scraping with template autoupdate capabilities

Implementasi Web Scraping Pada Sistem Informasi Lembaga Sertifikasi Profesi Universitas Siliwangi

Web Scraping with HTML DOM Method for Data Collection of Scientific Articles from Google Scholar

Contact Info

Product

Resources

About