GRACC: GRid ACcounting Collector

Abstract

The OSG has long maintained a central accounting system called Gratia. It uses small probes on each computing and storage resource in order to collect resource usage. The probes report to a central collector which stores the usage in a database. The database is then queried to generate reports. As the OSG aged, the size of the database grew very large. It became too large for the database technology to efficiently query to generate detailed reports. The design of a replacement requires data storage that could be queried efficiently to generate multi-year reports. Additionally, it requires flexibility to add new attributes to the collected data. In this paper we will describe updates to the GRACC architecture in the last 18 months. GRACC uses modern web technologies that were designed for large data storage, query, and visualization. That includes the open source database Elasticsearch, message broker software RabbitMQ, and Grafana and Kibana as data visualization platforms. It uses multiple agents that perform operations on the data to transform it for easier querying and summarization. *

Cosmos : A Unified Accounting System both for the HTCondor and Slurm Clusters at IHEP

“…Many systems [1][2][3][4][5] are developed to account resource usage. These systems have similar functional components but different implementations.…”

“…This is because although they share the same target of accounting resource usage, but they differ in infrastructure scale, accounting metrics and the degree of system complexity. Take CAOS [2] and GRACC [3] for instance. CAOS is developed to account cloud resource usage, and adopts layered architecture as well.…”