Benchmarking Private Cloud Performance with User-Centric Metrics

Abstract

Cloud computing is a new paradigm for the delivery of IT services. It has enabled many promising opportunities for features that cannot be easily implemented in traditional IT environments, such as elastic scalability, self-service deployment, resiliency and recovery, and so forth. Benchmarking the cloud requires a welldefined set of cloud performance metrics that should be able to sensitively distinguish the capabilities of cloud systems that enable those features. One way of defining benchmark metrics is based on observations of the internal mechanisms in a cloud. For example, an elasticity evaluation may be based on measuring a resource provisioning interval in the cloud. However, a more meaningful evaluation should be based on user-centric metrics. In this article, we will introduce a set of performance metrics that can be directly measured, calculated and compared by the cloud users, including workload consumers and the users who deploy and manage the workload life cycles. We will also discuss ways to organize the usercentric metrics, with different emphasis, into a benchmark that represents different use cases.

Cross-Layer Multi-Cloud Real-Time Application QoS Monitoring and Benchmarking As-a-Service Framework

“…In cloud environments, traditional benchmarking approaches cannot serve the users' needs [35]. Besides runtime performance, cloud specific attributes such as elasticity, deployment, resiliency, and recovery are required to be reflected in benchmarking process [35].…”

“…PaaS and SaaS) [36]. Despite of metrics Cloud-Cmp [37] can measure, authors in [35] stated that some of the metrics provided by CloudCmp are too experimental to be meaningful to cloud user, e.g. time to consistency.…”

“…In cloud environments, traditional benchmarking approaches cannot serve the users' needs [35]. Besides runtime performance, cloud specific attributes such as elasticity, deployment, resiliency, and recovery are required to be reflected in benchmarking process [35]. Further, benchmarking applications distributed in multicloud environments is a complex task as each application requires evaluation of distinct QoS metrics from others in order to evaluate the targeted cloud performance.…”