A Unified Monitoring Framework for Energy Consumption and Network Traffic

Abstract

Providing experimenters with deep insight about the effects of their experiments is a central feature of testbeds. In this paper, we describe Kwapi, a framework designed in the context of the Grid'5000 testbed, that unifies measurements for both energy consumption and network traffic. Because all measurements are taken at the infrastructure level (using sensors in power and network equipment), using this framework has no dependencies on the experiments themselves. Initially designed for OpenStack infrastructures, the Kwapi framework allows monitoring and reporting of energy consumption of distributed platforms. In this article, we present the extension of Kwapi to network monitoring, and outline how we overcame several challenges: scaling to a testbed the size of Grid'5000 while still providing high-frequency measurements; providing long-term loss-less storage of measurements; handling operational issues when deploying such a tool on a real infrastructure.

Energy monitoring as an essential building block towards sustainable ultrascale systems

“…Experiments have also shown that by exploiting features of monitoring hardware, KWAPI can monitor a large set of resources without perturbing experiments [46]. The throughput, latency, and jitter of measuring power consumption and transferring the measurement data have also been evaluated under several conditions [45].…”

“…The KWAPI framework 3 was created to enable measuring the power consumed by cluster resources in OpenStack [45]. It has been adapted to support largescale federations of clusters [46] and is currently the solution deployed on the Grid5000 platform [47] to monitor the energy consumed by computing and network resources. Its architecture, depicted in Figure 3, relies on a layer of drivers that retrieve power measurements from several devices, and plug-ins or data consumers that collect and process the measurement data.…”

“…The KWAPI framework has been deployed as a telemetry solution in Grid5000 [47], which comprises several sites with multiple clusters. The framework collects information of both power consumption and network usage, enabling users to evaluate the performance of their experiments [46]. Even though KWAPI does not build energy consumption models itself, it provides users and developers with feedback on their optimisation choices and has hence been used for profiling experiments and building power consumption models that are incorporated in discrete-event simulators.…”

SeDuCe: a Testbed for Research on Thermal and Power Management in Datacenters

“…The web portal and the API are both written in Python and leverage the "Flask" micro web framework. The API component makes an extensive use of the Swagger framework 7 which automatises the generation of complete REST web services and their documentations from a single description file (written in JSON or YAML). This choice has enabled us to focus on the definition and the implementation of the API, by reducing the quantity of required boilerplate code.…”

“…On the latter challenge, many research efforts have been conducted [1] [2], most of them focusing on the implementation of on demand power management systems, such as Dynamic voltage scaling (DVFS) [3] [4] and vary-on vary-off (VOVO) [5] [6]. Some work has been made to extend existing scientific testbeds with power monitoring of experiments : for example Kwapi [7] enables researchers to track the power consumption of their experiments conducted on Grid'5000.…”