Achieving cost/performance balance ratio using tiered storage caching techniques: A case study with CephFS

Poat, M D

doi:10.1088/1742-6596/898/6/062022

Cited by 2 publications

(3 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Ceph was a first choice as we still support active Ceph clusters and have experience with deploying in the past. [7] EPJ…”

mentioning

confidence: 99%

Federated Access from DOE Labs to Distributed Storage in the EIC Era of Computing

Poat,

Lauret,

Rao

2024

EPJ Web of Conf.

View full text Add to dashboard Cite

The Electron Ion Collider (EIC) collaboration and future experiment is a unique scientific ecosystem within Nuclear Physics as the experiment starts right off as a crosscollaboration between Brookhaven National Lab (BNL) & Jefferson Lab (JLab). As a result, this muti-lab computing model tries at best to provide services accessible from anywhere by anyone who is part of the collaboration. While the computing model for the EIC is not finalized, it is anticipated that the computational and storage resources will be made accessible to a wide range of collaborators across the world. The use of federated ID seems to be a critical element to the strategy of providing such services, allowing seamless access to each lab site computing resources. However, providing Federated access to a Federated storage is not a trivial matter and has its share of technical challenges. In this contribution, we focus on the steps we took towards the deployment of a distributed object storage system that integrates with Amazon S3 and Federated ID. We will first cover for and explain the first stage storage solutions provided to the EIC during the detector design phase. Our initial test deployment consisted of Lustre storage using MinIO, hence providing an S3 interface. High Availability load balancers were added later to provide the initial scalability it lacked. Performance of that system will be shown. While this embryonic solution worked well, it had many limitations. Looking ahead, the Ceph object storage is considered a top-of-the-line solution in the storage community - since the Ceph Object Gateway is compatible with the Amazon S3 API out of the box, our next phase will use a native S3 storage. Our Ceph deployment will consist of erasure coded storage nodes to maximize storage potential along with multiple Ceph Object Gateways for redundant access. We will compare performance of our next stage implementations. Finally, we will present how to leverage OpenID Connect with the Ceph Object Gateway’s to enable Federated ID access. We hope this contribution will serve the community needs as we move forward with cross-lab collaborations and the need for Federated ID access to distributed compute facilities.

show abstract

“…Ceph was a first choice as we still support active Ceph clusters and have experience with deploying in the past. [7] EPJ…”

mentioning

confidence: 99%

Federated Access from DOE Labs to Distributed Storage in the EIC Era of Computing

Poat,

Lauret,

Rao

2024

EPJ Web of Conf.

View full text Add to dashboard Cite

show abstract

“…Orientado a proveer almacenamiento a los clusters de cómputo de alto desempeño (HPC), sus componentes principales son los Servidores de Metadatos (MDS) , Metadata Targets (MDT), Servidores de Objetos (OSS), Object Server Targets (OST) y los clientes. Su principal característica es la capacidad de escalar exponencialmente, tanto en almacenamiento como en performance, y al estar orientado a clusters HPC, se adapta muy bien a cargas de trabajo en las que grandes archivos son accedidos de manera paralela por cientos de clientes de manera concurrente, aunque no tiene buen servicio para RESTFUL, S3 y Cloud Integration (Poat, 2017). Además, tanta potencia tiene sus costos: la implementación y gestión no es trivial, y no tiene la flexibilidad de uso que buscamos: ofrecer el acceso a nivel de archivos, objetos y volúmenes completos.…”

Section: Lustreunclassified

“…Ceph va a trabajar tan rápido como el más lento de sus componentes, dice Poat en su trabajo "Achieving Cost/Performance Balance Ratio using Tiered Storage Caching Techniques" (Poat, 2017) Entonces, ¿Cómo vamos a identificar el cuello de botella? ¿Qué vamos a medir?…”

Section: Benchmark De Los Discosunclassified

Estrategias de optimización y análisis de performance en sistemas de almacenamiento distribuido

Charne¹

View full text Add to dashboard Cite

La complejidad de los sistemas de almacenamiento es creciente dada la cantidad y simultaneidad de clientes conectados, los datos que son accedidos de manera concurrente, los usuarios distribuidos geográficamente, con tiempos de respuesta acotados y un volumen de información transferida que se amplía de forma exponencial. Respondiendo a estas necesidades, surgen soluciones de Almacenamiento Definido por Software (Software Defined Storage), en las que distintos dispositivos conectados a través de una red de datos forman un cluster que ofrece un conjunto de interfaces a las aplicaciones y clientes, y proveen un complejo sistema de gestión, mantenimiento y monitoreo de los distintos componentes. La tesis centra su análisis en Ceph, un sistema de almacenamiento distribuido de código abierto que corre sobre hardware genérico, diseñado para proveer escalabilidad, confiabilidad y alta performance. Se determinan metodología y métricas para el análisis de performance y se lleva adelante un procedimiento de optimización capa por capa, desde la interfaz de red y los discos, hasta las configuraciones en el cliente para lograr minimizar latencia y maximizar el througput. Finalmente, se estabece un modelo de performance a modo de "linea base", con el fin de monitorear el funcionamiento del cluster y utilizar como referencia en ajustes futuros.

show abstract

Achieving cost/performance balance ratio using tiered storage caching techniques: A case study with CephFS

Cited by 2 publications

References 2 publications

Federated Access from DOE Labs to Distributed Storage in the EIC Era of Computing

Federated Access from DOE Labs to Distributed Storage in the EIC Era of Computing

Estrategias de optimización y análisis de performance en sistemas de almacenamiento distribuido

Contact Info

Product

Resources

About