Web Data Extraction Approach for Deep Web using WEIDJ

Sabri, Ily Amalina Ahmad; Man, Mustafa; Bakar, Wan Azelee Wan Abu; Rose, Ahmad Nazari Mohd

doi:10.1016/j.procs.2019.12.124

Cited by 12 publications

(7 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Additionally, it should be mentioned that the Internet comprises of numerous superfluous components. Such redundant material on webpages (the so-called "noisy information") can greatly degrade the performance of the data extraction process (Ahmad Sabri et al, 2019). The key is to isolate and divest unnecessary information so that the collected data is as full as feasible in quality (Bartuś et al, 2017;Olszak & Zurada, 2020).…”

Section: Properties Of Data Acquisition Toolsmentioning

confidence: 99%

Reduction of information asymmetry in e-commerce: the web scraping approach

Hadasik

2024

View full text Add to dashboard Cite

The realm of science and its progress have always been built on research, and ultimately on the experience that precedes it. In order to arrive at scientific results appropriately, investigations must be conducted and those are done when data is accessible. In the modern era of digital world and society, data is gathered much simpler than before the widespread availability of computers and broadband networks. Sadly, it is a challenge for a beginner researcher to access databases gathered by various organizations since they are safeguarded and available only to a small audience (sometimes for an additional price). As data collecting becomes much simpler when we have access to IT solutions of the 21st century, it is even more convenient with a utilization of an application that automatically gathers and organizes data. Such an automized database building technique may become notably beneficial when we have a desire to collect unstructured data from a given period and from a specific website, in- dependently from the industry. This is where web scraping – a strategy that includes obtaining data from websites, is handy. In actuality, data extraction (especially approaches linked to the very web scraping) comprises of a large variety of distinct methods and technologies, such as data analysis, natural language syntax analysis, and information security. To get the most out of their advantages, it is of paramount importance to understand how they function. The role of information in the purchasing process has been extensively de- scribed in the literature. In doing so, attention was often drawn to the problem of information asymmetry – when the individual customer is informationally in a weaker position than the seller. This problem becomes particularly important in online shopping. The purpose of this work is to create an automated tool based on the web scraping technique that is designed to reduce the infor- mation asymmetry occurring in the buyer-seller relationship. The plane for de- picting the phenomenon of information asymmetry and the established web scraping tool is the automotive sector, with a particular focus on the essence of classifieds portal as a platform for matching buyers with sellers. The case of the largest automotive classifieds portal in Poland, which is OTOMOTO, was used in this study. The theoretical backdrop of this research, which serves as its begin- ning point, will be the problem of the uncertainty of judgments, coming from information asymmetry, an example of which is described in the groundbreak- ing essay by Akerlof (1970). In this work, the baseline environment for illustrating the problem of information asymmetry is also the automotive industry. In order to achieve the goal of this study, the following research questions were posed: RQ1. What are the implications of information asymmetry for judgment uncer- tainty in online transactions, and how can they be mitigated? RQ2. How can web scraping tools be designed to specifically address the chal- lenges of information asymmetry in the e-commerce sector? RQ3. What is the potential impact of reducing information asymmetry through web scraping on the overall efficiency and fairness of the e-commerce market, especially in automotive industry? This book is organized as follows. Chapter 1 outlines the theoretical back- ground with specific attention dedicated to the issue of information asymmetry as articulated in Akerlof (1970). Chapter 2 discusses the theoretical foundation of data extraction from internet resources (with particular focus on web scrap- ing), their characteristics, particularly legal as well as ethical issues, and the necessity to deploy data collection technologies in the research setting. In Chapter 3, a tool for data extraction created together with a suitable database to be able to harvest data from the OTOMOTO advertising site is discussed. The Chapter also provides technical elements including the Python language upon which the constructed tool is predicated. Chapter 3 additionally covers a practi- cal portion of the research in which a sample evaluation of the automotive in- dustry in Poland is done, which draws on the data gathered from OTOMOTO advertisement portal with the assistance of the built web scraping tool. The book can be found useful for researchers, academics, and data scien- tists, offering scholarly insights into reducing information asymmetry in e-commerce through web scraping. E-commerce practitioners and business owners in the automotive sector can gain competitive advantages by applying the book’s practical guidance for market analysis. The employment of the cre- ated web scraping tool, once quantitative data is retrieved, can be used by, e.g., data analysts, for the advanced analysis of the particular market, the verifica- tion of research hypotheses and the facilitation of decision-making processes. Policy makers, regulators, and legal professionals will find valuable perspectives on the legal implications of web scraping in enhancing information transparency. On the other hand, everyday customers of online stores may benefit from the theoretical and practical value that this book brings, especially with their will- ingness to compare offers posted in advertisements, further analyze them, and make the right purchase decision for themselves based on more complete access to information (or put another way: minimized uncertainty among buyers). This monograph is an adaptation of the author’s master’s thesis with the same title, which was defended in July 2021 at the University of Economics in Katowice (Poland) under the supervision of Associate Professor Maria Mach- -Król, PhD. The thesis was defended with a very good result and served as the basis for issuing a Master’s degree diploma with distinction to the author. The thesis was awarded the second degree prize in the 2022 nationwide competi- tion of diploma theses in the field of economic informatics, which was awarded by the Scientific Society of Economic Informatics (Częstochowa, Poland).

show abstract

Section: Properties Of Data Acquisition Toolsmentioning

confidence: 99%

Reduction of information asymmetry in e-commerce: the web scraping approach

Hadasik

2024

View full text Add to dashboard Cite

show abstract

“…Merujuk Kamus Besar Bahasa Indonesia, istilah "web" sudah diserap ke dalam bahasa Indonesia yang artinya adalah "sistem untuk mengakses, memanipulasi, dan mengunduh dokumen hipertaut yang terdapat dalam komputer yang dihubungkan melalui internet" [23]. Web data extraction adalah penggunaan perangkat teknologi untuk secara otomatis mengekstrasi dan mengorganisasikan data yang berasal dari web sehingga kemudian data tersebut dapat dianalisis lebih lanjut [24]. Beberapa istilah lain yang sering dipertukarkan dengan istilah ini adalah web scraping, web crawling ataupun web harvesting.…”

Section: Web Data Extractionunclassified

Perancangan Model Sistem E-Working Paper Berbasis Web Untuk Peningkatan Kualitas Proses Audit

2023

View full text Add to dashboard Cite

Audit memiliki peranan yang sangat signifikan dalam mengontrol proses bisnis di berbagai sektor. Dalam artikel ini permasalahan utama dari proses audit adalah masih rendahnya efisiensi dan efektifitas praktek audit saat ini di Indonesia. Keterbatasan teknologi informasi menjadi satu kunci utama yang menyebabkan kurang maksimalnya proses audit. Masih kurangnya ditemukan pengembangan sistem audit berbasis digital dan terintegrasi yang dapat diterapkan di Indonesia. Permasalahan seperti mahalnya biaya lisensi sebuah aplikasi sistem audit dari luar negeri akhirnya memaksa banyak auditor untuk bertahan menggunakan sistem audit berbasis dekstop yang sangat terbatas. Oleh karena itu, artikel ini memberikan analisa model untuk mengembangkan sebuah sistem kertas kerja audit elektronik berbasis WEB yang akan meningkatkan kualitas proses audit di Indonesia tanpa membutuhkan biaya lisensi yang tinggi. Hasil penelitian berupa rincian usulan kerangka, model hingga rencana kerja untuk merancang kertas kerja audit elektronik berbasis WEB yang dapat diaplikasikan secara langsung.

show abstract

“…Aspek ini perlu mendapatkan perhatian dan pertimbangan dalam melakukan web data extraction. Beberapa pendapat terkait ini (lihat misalnya: [16,41,42,43]) web data extraction harus dilakukan dengan membaca syarat dan ketentuan yang dinyatakan oleh pemilik situs atau laman web. Hal ini untuk menjaga munculnya risiko tuntutan hukum dari pengelola situs atau laman web yang datanya diekstraksi.…”

Section: B Web Data Extractionunclassified

“…Teknik ini jika dibandingkan metode pertama akan lebih mudah jika auditor melakukan ekstraksi data terhadap banyak laman web karena teknik ini memungkinkan ada iterasi atas perhatian pengambilan data dengan URL yang berubah-ubah sesuai dengan parameter yang dimasukkan ke dalam URL. Dengan catatan, teknik ini tidak bisa digunakan terhadap situs web yang menyimpan parameter pencarian dalam URL-nya dengan metode POST atau laman web yang mengandung CSS maupun JavaScript (lihat misalnya: [41,42]). Secara teknis, cara ini memerlukan beberapa langkah untuk mengubah script yang telah dibuatkan secara otomatis oleh Power Query dengan cara menambahkan Invoke Custom Function dan membuat sebuah Table yang isinya adalah parameter tanggal kurs pajak sesuai dengan rentang waktu yang diinginkan.…”

Section: Eksperimen Dan Analisisunclassified

“…Tabel 2 menyajikan perbandingan antara ketiga teknik web data extraction yang sudah diuraikan di atas. Selain perangkat yang dicontohkan ini tentu saja masih sangat banyak sekali perangkat yang beredar dan ditawarkan oleh para vendornya ataupun hasil penelitian yang sudah dipublikasikan (lihatnya misalnya [42]). Pemilihan perangkat web data extraction dalam penelitian ini dilakukan oleh penulis secara arbitrer dan memang secara praktik nyata penulis telah menggunakan perangkat tersebut untuk pelaksanaan ADA.…”

Section: Eksperimen Dan Analisisunclassified

See 1 more Smart Citation

Web Data Extraction Dalam Analitika Data Audit: Pengembangan Artefak Teknologi Dalam Perspektif Design Science Research

Darono

2020

teknika

View full text Add to dashboard Cite

Perkembangan implementasi Teknologi Informasi dan Komunikasi (TIK) sebagai bagian pengendalian internal organisasi mendorong auditor mengembangkan analitika data audit (ADA/Audit Data Analytics) sebagai kerangka pengetahuan dan praktik untuk mendapatkan bukti audit dan informasi lainnya dari sekumpulan data elektronik terkait dengan pelaksanaan pada semua tahapan pekerjaan audit. Pada saat yang sama, terdapat kecenderungan organisasi untuk menyajikan datanya dengan aplikasi berbasis web. Terkait dengan keberadaan laman web sebagai sumber data (bukti audit) tersebut, telah berkembang teknik ekstraksi data dari laman web yang disebut dengan web data extraction. Penelitian ini dengan menggunakan design science research methodology mengajukan temuan artefak yang berkaitan dengan model dan instantiasi (instantiation) web data extraction untuk implementasi ADA. Hasil penelitian ini diharapkan dapat menjadi tambahan referensi dalam ranah praktik audit berupa artefak dalam bentuk instantiasi penggunaan web data extraction untuk akusisi data sebagai bukti audit dengan sumber dari halaman web, baik dari aplikasi berbasis intranet ataupun internet. Penelitian ini juga berkontribusi dengan mengajukan kerangka praktikal implementasi web data extraction sebagai bagian dari ADA dalam melaksanakan pekerjaan audit. Selain itu, hasil kajian ini juga diharapkan menjadi referensi untuk penggunaan design science research methodology yang ternyata belum terlalu banyak diaplikasikan dalam penelitian dalam disiplin audit di Indonesia.

show abstract

Web Data Extraction Approach for Deep Web using WEIDJ

Cited by 12 publications

References 25 publications

Reduction of information asymmetry in e-commerce: the web scraping approach

Reduction of information asymmetry in e-commerce: the web scraping approach

Perancangan Model Sistem E-Working Paper Berbasis Web Untuk Peningkatan Kualitas Proses Audit

Web Data Extraction Dalam Analitika Data Audit: Pengembangan Artefak Teknologi Dalam Perspektif Design Science Research

Contact Info

Product

Resources

About