A prototype for the evolution of ATLAS EventIndex based on Apache Kudu storage

Baranowski, Zbigniew; Canali, Luca; Casaní, Álvaro Fernández; Montoro, Carlos García; Hoz, S. González de la; Hřivnáć, J.; Prokoshin, F.; Rybkine, G.; Sánchez, J.; Barberis, D.

doi:10.1051/epjconf/201921404057

Cited by 4 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Investigations on several structured storage formats for the main EventIndex data to replace the Hadoop MapFiles started a few years ago [17]. Initially it looked like Apache Kudu [18] would be a good solution, joining BigData storage performance with SQL query capabilities [19]. Unfortunately Kudu did not get a large support in the open-source community and CERN decided not to invest hardware and manpower resources in this technology.…”

Section: System Design Evolutionmentioning

confidence: 99%

The ATLAS EventIndex for LHC Run 3

et al. 2020

Self Cite

View full text Add to dashboard Cite

The ATLAS EventIndex was designed in 2012-2013 to provide a global event catalogue and limited event-level metadata for ATLAS analysis groups and users during the LHC Run 2 (2015-2018). It provides a good and reliable service for the initial use cases (mainly event picking) and several additional ones, such as production consistency checks, duplicate event detection and measurements of the overlaps of trigger chains and derivation datasets. The LHC Run 3, starting in 2021, will see increased data-taking and simulation production rates, with which the current infrastructure would still cope but may be stretched to its limits by the end of Run 3. This proceeding describes the implementation of a new core storage service that will be able to provide at least the same functionality as the current one for increased data ingestion and search rates, and with increasing volumes of stored data. It is based on a set of HBase tables, with schemas derived from the current Oracle implementation, coupled to Apache Phoenix for data access; in this way we will add to the advantages of a BigData based storage system the possibility of SQL as well as NoSQL data access, allowing to re-use most of the existing code for metadata integration.

show abstract

Section: System Design Evolutionmentioning

confidence: 99%

The ATLAS EventIndex for LHC Run 3

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…This improves performance and alsoavailability, as data is also replicated in several tablets on different servers. A careful key and partition schema definition for the EventIndex is being designed [5], with the aim to allow that row entries that define different reprocessings of the same data sit in the same partition and close in disk storage. This fact improves navigation through locality, and also benefits from better compression ratios of the same data due to this locality.…”

Section: New Backend Solution: Kudumentioning

confidence: 99%

“…More details about the implementation on Kudu and the tests that we have been performing related with the Data Collection task can be seen in Ref. [5].…”

Section: New Backend Solution: Kudumentioning

confidence: 99%

Distributed Data Collection for the Next Generation ATLAS EventIndex Project

et al. 2019

Self Cite

View full text Add to dashboard Cite

The ATLAS EventIndex currently runs in production in order to build a complete catalogue of events for experiments with large amounts of data. The current approach is to index all final produced data files at CERN Tier0, and at hundreds of grid sites, with a distributed data collection architecture using Object Stores to temporarily maintain the conveyed information, with references to them sent with a Messaging System. The final backend of all the indexed data is a central Hadoop infrastructure at CERN; an Oracle relational database is used for faster access to a subset of this information. In the future of ATLAS, instead of files, the event should be the atomic information unit for metadata, in order to accommodate future data processing and storage technologies. Files will no longer be static quantities, possibly dynamically aggregating data, and also allowing event-level granularity processing in heavily parallel computing environments. It also simplifies the handling of loss and or extension of data. In this sense the EventIndex may evolve towards a generalized whiteboard, with the ability to build collections and virtual datasets for end users. This proceedings describes the current Distributed Data Collection Architecture of the ATLAS EventIndex project, with details of the Producer, Consumer and Supervisor entities, and the protocol and information temporarily stored in the ObjectStore. It also shows the data flow rates and performance achieved since the new Object Store as temporary store approach was put in production in July 2017. We review the challenges imposed by the expected increasing rates that will reach 35 billion new real events per year in Run 3, and 100 billion new real events per year in Run 4. For simulated events the numbers are even higher, with 100 billion events/year in run 3, and 300 billion events/year in run 4. We also outline the challenges we face in order to accommodate future use cases in the EventIndex.

show abstract

“…Investigations on several structured storage formats for the main EventIndex data to replace the Hadoop MapFiles [4] used till now started a few years ago [5]. Initially it looked like Apache Kudu [6] would be a good solution, as it joins BigData storage performance with SQL query capabilities [7]. Unfortunately Kudu did not get a sufficiently large support in the open-source community and CERN decided not to invest hardware and human resources in this technology.…”

Section: Introductionmentioning

confidence: 99%

“…Proceedings of the 9th International Conference "Distributed Computing and Grid Technologies in Science andEducation" (GRID'2021), Dubna, Russia, July[5][6][7][8][9] 2021 …”

mentioning

confidence: 99%

The Atlas Eventindex Using the Hbase/Phoenix Storage Solution

Cherepanova¹,

Alexandrov²,

Aleksandrov³

et al. 2021

9th International Conference "Distributed Computing and Grid Technologies in Science and Education"

View full text Add to dashboard Cite

The ATLAS EventIndex provides a global event catalogue and event-level metadata for ATLAS analysis groups and users. The LHC Run 3, starting in 2022, will see increased data-taking and simulation production rates, with which the current infrastructure would still cope but may be stretched to its limits by the end of Run 3. This talk describes the implementation of a new core storage service that will provide at least the same functionality as the current one for increased data ingestion and search rates, and with increasing volumes of stored data. It is based on a set of HBase tables, coupled to Apache Phoenix for data access; in this way we will add to the advantages of a BigData based storage system the possibility of SQL as well as NoSQL data access, which allows the re-use of most of the existing code for metadata integration.

show abstract

A prototype for the evolution of ATLAS EventIndex based on Apache Kudu storage

Cited by 4 publications

References 7 publications

The ATLAS EventIndex for LHC Run 3

The ATLAS EventIndex for LHC Run 3

Distributed Data Collection for the Next Generation ATLAS EventIndex Project

The Atlas Eventindex Using the Hbase/Phoenix Storage Solution

Contact Info

Product

Resources

About