A Clustering Model Based on Matrix Approximation with Applications to Cluster System Log Files

Abstract

Abstract. In system management applications, to perform automated analysis of the historical data across multiple components when problems occur, we need to cluster the log messages with disparate formats to automatically infer the common set of semantic situations and obtain a brief description for each situation. In this paper, we propose a clustering model where the problem of clustering is formulated as matrix approximations and the clustering objective is minimizing the approximation error between the original data matrix and the reconstructed matrix based on the cluster structures. The model explicitly characterizes the data and feature memberships and thus enables the descriptions of each cluster. We present a two-side spectral relaxation optimization procedure for the clustering model. We also establish the connections between our clustering model with existing approaches. Experimental results show the effectiveness of the proposed approach.

End-to-end framework for fault management for open source clusters

“…A survey of analysis methods for system logs covering up to 2005 is given Vaarandi [35]. There is an extensive literature on statistical fault detection [2], detection using clustering [34], [16], [21], and fault prediction [7] and [27].…”

Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions

“…In that sense, our methods are somewhat more flexible than the double k-means. Li and Peng (2005) relaxed the hard double k-means method using orthogonal constrained factor matrices instead of indicator matrices. However, they did not impose nonnegativity constraints on these matrices.…”