A study of data representation in Hadoop to optimize data storage and search performance for the ATLAS EventIndex

Baranowski, Zbigniew; Canali, Luca; Toebbicke, R.; Hřivnáć, J.; Barberis, D.

doi:10.1088/1742-6596/898/6/062020

Cited by 7 publications

(6 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For operations using trigger information, first we get the dspid from a particular entry of the Datasets table with query (4), then we use the previously retrieved dspid to select all the events from this dataset that also have (for example) the trigger 100 on the TAV (trigger after veto) mask, as in query (5). On the pre-production system this query lasted 107 seconds on the first batch, and 96 seconds on the second batch, but on the test machine with less data and no other users this test lasted only 12 seconds.…”

Section:  Events Tablementioning

confidence: 99%

See 1 more Smart Citation

The Atlas Eventindex Using the Hbase/Phoenix Storage Solution

Cherepanova¹,

Alexandrov²,

Aleksandrov³

et al. 2021

9th International Conference "Distributed Computing and Grid Technologies in Science and Education"

Self Cite

View full text Add to dashboard Cite

The ATLAS EventIndex provides a global event catalogue and event-level metadata for ATLAS analysis groups and users. The LHC Run 3, starting in 2022, will see increased data-taking and simulation production rates, with which the current infrastructure would still cope but may be stretched to its limits by the end of Run 3. This talk describes the implementation of a new core storage service that will provide at least the same functionality as the current one for increased data ingestion and search rates, and with increasing volumes of stored data. It is based on a set of HBase tables, coupled to Apache Phoenix for data access; in this way we will add to the advantages of a BigData based storage system the possibility of SQL as well as NoSQL data access, which allows the re-use of most of the existing code for metadata integration.

show abstract

Section:  Events Tablementioning

confidence: 99%

“…Investigations on several structured storage formats for the main EventIndex data to replace the Hadoop MapFiles [4] used till now started a few years ago [5]. Initially it looked like Apache Kudu [6] would be a good solution, as it joins BigData storage performance with SQL query capabilities [7].…”

Section: Introductionmentioning

confidence: 99%

The Atlas Eventindex Using the Hbase/Phoenix Storage Solution

Cherepanova¹,

Alexandrov²,

Aleksandrov³

et al. 2021

9th International Conference "Distributed Computing and Grid Technologies in Science and Education"

Self Cite

View full text Add to dashboard Cite

show abstract

“…Investigations on several structured storage formats for the main EventIndex data to replace the Hadoop MapFiles started a few years ago [17]. Initially it looked like Apache Kudu [18] would be a good solution, joining BigData storage performance with SQL query capabilities [19].…”

Section: System Design Evolutionmentioning

confidence: 99%

The ATLAS EventIndex for LHC Run 3

et al. 2020

View full text Add to dashboard Cite

The ATLAS EventIndex was designed in 2012-2013 to provide a global event catalogue and limited event-level metadata for ATLAS analysis groups and users during the LHC Run 2 (2015-2018). It provides a good and reliable service for the initial use cases (mainly event picking) and several additional ones, such as production consistency checks, duplicate event detection and measurements of the overlaps of trigger chains and derivation datasets. The LHC Run 3, starting in 2021, will see increased data-taking and simulation production rates, with which the current infrastructure would still cope but may be stretched to its limits by the end of Run 3. This proceeding describes the implementation of a new core storage service that will be able to provide at least the same functionality as the current one for increased data ingestion and search rates, and with increasing volumes of stored data. It is based on a set of HBase tables, with schemas derived from the current Oracle implementation, coupled to Apache Phoenix for data access; in this way we will add to the advantages of a BigData based storage system the possibility of SQL as well as NoSQL data access, allowing to re-use most of the existing code for metadata integration.

show abstract

“…In the meantime BigData technologies advanced and now we have the choice between many different products and options. Studies of new data formats and/or new storage technologies [5] concluded that Kudu is the most promising technology for the next few years. Hence this prototype.…”

Section: Motivation For the Project Evolutionmentioning

confidence: 99%

A prototype for the evolution of ATLAS EventIndex based on Apache Kudu storage

et al. 2019

View full text Add to dashboard Cite

The ATLAS EventIndex has been in operation since the beginning of LHC Run 2 in 2015. Like all software projects, its components have been constantly evolving and improving in performance. The main data store in Hadoop, based on MapFiles and HBase, can work for the rest of Run 2 but new solutions are explored for the future. Kudu offers an interesting environment, with a mixture of BigData and relational database features, which look promising at the design level. This environment is used to build a prototype to measure the scaling capabilities as functions of data input rates, total data volumes and data query and retrieval rates. In this proceedings we report on the selected data schemas and on the current performance measurements with the Kudu proto-

show abstract

A study of data representation in Hadoop to optimize data storage and search performance for the ATLAS EventIndex

Cited by 7 publications

References 6 publications

The Atlas Eventindex Using the Hbase/Phoenix Storage Solution

The Atlas Eventindex Using the Hbase/Phoenix Storage Solution

The ATLAS EventIndex for LHC Run 3

A prototype for the evolution of ATLAS EventIndex based on Apache Kudu storage

Contact Info

Product

Resources

About