Towards Predictive Maintenance with Machine Learning at the INFN-CNAF computing centre

Abstract

The INFN-CNAF computing center, one of the Worldwide LHC Computing Grid Tier-1 sites, is serving a large set of scientific communities, in High Energy Physics and beyond. In order to increase efficiency and to remain competitive in the long run, CNAF is launching various activities aiming at implementing a global predictive maintenance solution for the site. This requires a site-wide effort in collecting, cleaning and structuring all possibly useful data coming from log files of the various Tier-1 services and systems, as a necessary step prior to designing machine learning based approaches for predictive maintenance. Among the Tier-1 services, efficient storage systems are one of the key ingredients of Tier-1 operations. CNAF uses the StoRM service as a Grid Storage Resource Manager solution: its operations are logged in a very complex manner, as the log content is deeply unstructured and hard to be exploited for analytics purposes. Despite such difficulty, the StoRM logs are a precious source of information for operators (e. g. real-time monitoring and anomaly detection), for developers (e. g. debugging, service stability, code improvements) and for site managers (service optimization, storage usage efficiency, time and money saving ways to spot and prevent unwanted behaviors). Based on previous experiences on Big Data Analytics and Machine/Deep learning in the CMS experiment, this work describes how the StoRM logs can be handled and parsed to extract the relevant information, how such log handling can be designed to work automatically, how to define and implement metrics to tag critical states of the service, how to correlate StoRM events with external services events, and ultimately how to contribute to the future CNAF-wide predictive maintenance system. Initial results in this activity are presented and discussed. Furthermore, a mention to ongoing complementary work at the CNAF center is also mentioned.

Operational Intelligence for Distributed Computing Systems for Exascale Science

“…Currently, both supervised and unsupervised approaches are used [18], as well as anomaly detection and log template extraction approaches [19], and various techniques adopted from Natural Language Processing (NLP) research. All methods are based on a quick prototype and validation cycle, in order to timely develop, train, and test models against new unseen log data being collected in real time, with the goal of validating one or more approaches as valuable to the early detection of symptoms of future failures at sites.…”

Detecção de Anomalias em Logs para Manutenção Preditiva baseada em Sistema Fuzzy Evolutivo Fracamente Supervisionado

“…Um primeiro trabalho baseado em Elastic Stack Suite cataloga os registros de log e as anomalias usando uma ferramenta de aprendizado de máquina não-supervisionado (Diotalevi et al, 2019). Outra iniciativa usa abordagens supervisionadas de inteligência computacional para prever anomalias no comportamento do sistema em uma solução ad-hoc (Giommi et al, 2019). Em (Minarini, 2019) foi criado um protótipo para diferenciar comportamentos usuais e anômalos do sistema via máquinas de vetor de suporte.…”

Comparison of Evolving Granular Classifiers applied to Anomaly Detection for Predictive Maintenance in Computing Centers

“…Many efforts have been done to maintain the quality of service (QoS) of the WLCG. In particular, at the Tier-1 computing center at Bologna, supervised machine learning methods to predict anomalies of the StoRM service [13] have been considered by using ad-hoc data processing methods. Another initiative concerns a system based on the Elastic Stack Suite to collect, parse and catalogue log data, as well as classifying anomalies using an embedded unsupervisedlearning tool [14].…”