Data intensive ATLAS workflows in the Cloud

Abstract

Large physics experiments, such as ATLAS, have participating physicists and institutes all over the Globe. Nowadays, physics analyses are performed on data that is stored thousands of kilometres away. This is possible due to the distributed computing infrastructure known as the Worldwide LHC Computing Grid (WLCG). In addition to the analyses, all the previous data transformation steps, such as raw data reconstruction, are performed within the WLCG. Within the next decade, the computing requirements are projected to exceed the available resources by a factor of ten. In order to mitigate this discrepancy, alternative computing solutions have to be investigated. Within this thesis, the viability of Cloud computing is evaluated. The concept of Cloud computing is to rent infrastructure from a commercial provider. In contrast to that, in the WLCG computing concept the hardware within the computing centres is purchased and operated by the WLCG. In order to examine Cloud computing, a model that predicts the workflow performance on a given infrastructure is created, validated and applied. In parallel, the model was used to evaluate a workflow optimisation technique called overcommitting. Overcommitting means that the workload on a computer consists of more parallel processes than there are CPU cores. This technique is used to fill otherwise idle CPU cycles and thereby increase the CPU utilisation. Using the model, overcommitting is determined to be a viable optimisation technique, especially when using remote data input, taking into account the increased memory footprint. Introducing the overcommitting considerations to the Cloud viability evaluation increases the feasibility of Cloud computing. This is because Cloud computing may not include a storage solution and has the flexibility to provision virtual machines with additional memory. The final conclusion is drawn by taking the above described results and by combining them with the cost of the WLCG and the Cloud. The result is that Cloud computing is not yet competitive compared to the WLCG computing concept. Data intensive ATLAS workflows in the CloudZusammenfassung Die großen Physikexperimente, wie zum Beispiel ATLAS, bestehen aus Kollaborationen mit Physikern und Instituten auf der ganzen Welt. Heutzutage werden physikalische Analysen an Daten durchgeführt, die Tausende von Kilometern entfernt gespeichert sind. Dies ist aufgrund der verteilten Computing-Infrastruktur, die als Worldwide LHC Computing Grid (WLCG) bekannt ist, möglich. Zusätzlich zu den Analysen werden alle vorherigen Datentransformationsschritte, wie die Rekonstruktion von Rohdaten, innerhalb des WLCG durchgeführt. Innerhalb des nächsten Jahrzehnts wird erwartet, dass die Anforderungen an die Computerinfrastruktur die verfügbaren Ressourcen um den Faktor zehn übersteigen werden. Um diese Diskrepanz zu mindern, müssen Alternativen zur jetzigen Computerinfrastruktur untersucht werden. Im Rahmen dieser Arbeit wird Cloud Computing evaluiert. Das Konzept von Cloud Computing besteht darin, eine Computerinfr...