Towards Representative Web Performance Measurements with Google Lighthouse

Abstract

Web performance testing with tools such as Google Lighthouse is a common task in software practice and research. However, variability in time-based performance measurement results is observed quickly when using the tool, even if the website has not changed. This can occur due to variability in the network, web, and client devices. In this paper, we investigated how this challenge was addressed in the existing literature. Furthermore, an experiment was conducted, highlighting how unrepresentative measurements can result from single runs; thus, researchers and practitioners are advised to run performance tests multiple times and use an aggregation value. Based on the empirical results, 5 consecutive runs using a median to aggregate results reduce variability greatly, and can be performed in a reasonable time. The study’s findings alert to p otential pitfalls when using single run-based measurement results and serve as guidelines for future use of the tool.

Towards Improving Accessibility of Web Auditing with Google Lighthouse

“…This is because the tool is able to handle a single input, mainly in json format and comes with limited output type when producing the performance reports of web analysis. There has been a growing need for the Google Lighthouse to support multiple input types and report output [3]. Also, the unnecessary time it takes the Google Lighthouse in the running the report of a single page from a website rather the whole pages of a website is considered as one of the issues resulting from weak accessibility and performance.…”