When the testing gets tough, the tough get ElasTest

Bertolino, Antonia; Calabrò, Antonello; Angelis, Guglielmo De; Gallego, Micael; García, Boni; Gortázar, Francisco

doi:10.1145/3183440.3183497

Cited by 14 publications

(8 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Depending on the hardware and the SUT, this fact might lead to flaky tests [41]. Regarding troubleshooting, observability tools like the ELK (ElasticSearch, Logstash, and Kibana) stack [42] or ElasTest (https://elastest.io/) [43] could help alleviate this problem.…”

Section: Discussionmentioning

confidence: 99%

A Survey of the Selenium Ecosystem

et al. 2020

Self Cite

View full text Add to dashboard Cite

Selenium is often considered the de-facto standard framework for end-to-end web testing nowadays. It allows practitioners to drive web browsers (such as Chrome, Firefox, Edge, or Opera) in an automated fashion using different language bindings (such as Java, Python, or JavaScript, among others). The term ecosystem, referring to the open-source software domain, includes various components, tools, and other interrelated elements sharing the same technological background. This article presents a descriptive survey aimed to understand how the community uses Selenium and its ecosystem. This survey is structured in seven categories: Selenium foundations, test development, system under test, test infrastructure, other frameworks, community, and personal experience. In light of the current state of Selenium, we analyze future challenges and opportunities around it.

show abstract

Section: Discussionmentioning

confidence: 99%

A Survey of the Selenium Ecosystem

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…This paper has been done in the context of the ElasTest (https://elastest.io/) project, which is an H2020 European Union funded initiative. ElasTest has born with the big ambition of providing a comprehensive solution to ease the process of development, operation, and maintenance of end-to-end tests for different kind of applications, including web and mobile among others [20]. WebRTC is a relevant demonstrator technology in ElasTest, and thus, this work is a part of the research efforts in this line.…”

Section: Motivation and Contributionsmentioning

confidence: 99%

Practical Evaluation of VMAF Perceptual Video Quality for WebRTC Applications

et al. 2019

Self Cite

View full text Add to dashboard Cite

WebRTC is the umbrella term for several emergent technologies aimed to exchange real-time media in the Web. Like other media-related services, the perceived quality of WebRTC communication can be measured using Quality of Experience (QoE) indicators. QoE assessment methods can be classified as subjective (users’ evaluation scores) or objective (models computed as a function of different parameters). In this paper, we focus on VMAF (Video Multi-method Assessment Fusion), which is an emergent full-reference objective video quality assessment model developed by Netflix. VMAF is typically used to assess video streaming services. This paper evaluates the use of VMAF in a different type of application: WebRTC. To that aim, we present a practical use case built on the top of well-known open source technologies, such as JUnit, Selenium, Docker, and FFmpeg. In addition to VMAF, we also calculate other objective QoE video metrics such as Visual Information Fidelity in the pixel domain (VIFp), Structural Similarity (SSIM), or Peak Signal-to-Noise Ratio (PSNR) applied to a WebRTC communication in different network conditions in terms of packet loss. Finally, we compare these objective results with a subjective evaluation using a Mean Opinion Score (MOS) scale to the same WebRTC streams. As a result, we found a strong correlation of the subjective video quality perceived in WebRTC video calls with the objective results computed with VMAF and VIFp in comparison with SSIM and PSNR and their variants.

show abstract

“…FRAP [4]). ARRESTT [2] covers reproducible software testing methodologies, and Elastest [1] covers programmable end-to-end testing with failure injection in the cloud. Further techniques have been proposed to continuously monitor runtime and security properties of composite cloud applications but without capturing the observations as datasets [3].…”

Section: Related Workmentioning

confidence: 99%