On Observability and Monitoring of Distributed Systems – An Industry Interview Study

Abstract

Business success of companies heavily depends on the availability and performance of their client applications. Due to modern development paradigms such as DevOps and microservice architectural styles, applications are decoupled into services with complex interactions and dependencies. Although these paradigms enable individual development cycles with reduced delivery times, they cause several challenges to manage the services in distributed systems. One major challenge is to observe and monitor such distributed systems. This paper provides a qualitative study to understand the challenges and good practices in the field of observability and monitoring of distributed systems. In 28 semi-structured interviews with software professionals we discovered increasing complexity and dynamics in that field. Especially observability becomes an essential prerequisite to ensure stable services and further development of client applications. However, the participants mentioned a discrepancy in the awareness regarding the importance of the topic, both from the management as well as from the developer perspective. Besides technical challenges, we identified a strong need for an organizational concept including strategy, roles and responsibilities. Our results support practitioners in developing and implementing systematic observability and monitoring for distributed systems.

Data Fusion of Observability Signals for Assisting Orchestration of Distributed Applications

“…Based on an industrial survey on microservice tracing and analysis [16], distributed tracing and analysis is considered an important part of the infrastructure for industrial microservice systems, while the development of efficient data fusion mechanisms for trace analysis and production of business intelligence reports is considered challenging. In a similar qualitative study on identifying the challenges and good practices in the field of observability and monitoring of distributed systems [17], the main reported challenge was found to be the need for the management of heterogeneity in the adopted microservices-based development and deployment paradigms, since isolated monitoring and observability solutions are adopted even across teams within the same organization. Targeted studies on specific development paradigms are also provided, such as the study in [7] that focuses on testing and debugging of serverless-based applications.…”

On Analysis of Security and Elasticity Dependency in IIoT Platform Services

“…Similarly, an observability enhancing framework such as [10] discusses the feasibility only over cloud subsystem microservices. Systems such as [11], [12] for observability in IoT and edge networks usually do not integrate with cloud native applications and hence are not scalable onto the cloud subsystems [13]. The BonFIRE framework by Kavoussanakis et al [14] aims to create a testbed for cloud applications and allows the user to host wide scenarios of services that are not yet deployed to the cloud.…”

FaaSter Troubleshooting -- Evaluating Distributed Tracing Approaches for Serverless Applications

“…Most of the faults investigated relate to developer bugs in particular and not platform influences that are typical in serverless applications. More generally, Niedermaier et al [17] also survey challenges and good practices in the field of observability and monitoring of distributed systems.…”