Extracting usage patterns from web server log

Jeba, J. Monisha Privthy; Bhuvaneswari, M.; Muneeswaran, K.

doi:10.1109/icghpc.2016.7508074

Cited by 5 publications

(1 citation statement)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Log files are often automatically deleted from servers due to their constant increase in volume. There are various techniques for analyzing WSAL files on the market [4] [5], but the main problem lies in storing them in their raw form in order to allow various analysis processes to be run in the future enabling information to be extracted for different purposes. Transforming this logs into structured data for later storage, using traditional methods, would involve high storage costs.…”

Section: Introductionmentioning

confidence: 99%

Data Lake Architecture for Storing and Transforming Web Server Access Log Files

Zagan

Danubianu

2023

IEEE Access

View full text Add to dashboard Cite

Web server access log files are text files containing important data about server activities, client requests addressed to a server, server responses, etc. Large-scale analysis of these data can contribute to various improvements in different areas of interest. The main problem lies in storing these files in their raw form, over long time, to allow analysis processes to be run at any time enabling information to be extracted as foundation for high quality decisions. Our research focuses on offering an economical, secure, and highperformance solution for the storage of large amount of raw log files. Proposed system implements a Data Lake (DL) architecture in cloud using Azure Data Lake Storage Gen2 (ADLS Gen2) for extract-loadtransform (ELT) pipelines. This architecture allows large volumes of data to be stored in their raw form. Afterwards they can be subjected to transformation and advanced analysis processes without the need of a structured writing scheme. The main contribution of this paper is to provide a solution that is affordable and more accessible to perform web server access log data ingestion, storage and transformation over the newest technology, Data Lake. As derivative contribution, we proposed the use of Azure Blob Trigger Function to implement the algorithm of transforming log files into parquet files leading to 90% reduction in storage space compared to their original size. That means much lower storage costs than if they had been stored as log files.A hierarchical data storage model has also been proposed for shared access to data over different layers in the DL architecture, on top of which Data Lifecycle Management (DLM) rules have been proposed for storage cost efficiency. We proposed ingesting log files into a Data Lake deployed in cloud due to ease of deployment and low storage costs. The aim is to maintain this data in the long term, to be used in future advanced analytics processes by cross-referencing with other organizational or external data. That could bring important benefits. While the proposed solution is explicitly based on ADLS Gen2, it represents an important benchmark in approaching a cloud DL solution offered by any other vendor.

show abstract

Section: Introductionmentioning

confidence: 99%