A Big Data analyzer for large trace logs

Abstract

Current generation of Internet-based services are typically hosted on large data centers that take the form of warehouse-size structures housing tens of thousands of servers. Continued availability of a modern data center is the result of a complex orchestration among many internal and external actors including computing hardware, multiple layers of intricate software, networking and storage devices, electrical power and cooling plants. During the course of their operation, many of these components produce large amounts of data in the form of event and error logs that are essential not only for identifying and resolving problems but also for improving data center efficiency and management. Most of these activities would benefit significantly from data analytics techniques to exploit hidden statistical patterns and correlations that may be present in the data. The sheer volume of data to be analyzed makes uncovering these correlations and patterns a challenging task. This paper presents Big Data analyzer (BiDAl), a prototype Java tool for log-data analysis that incorporates several Big Data technologies in order to simplify the task of extracting information from data traces produced by large clusters and server farms. BiDAl provides the user with several analysis languages (SQL, R and Hadoop MapReduce) and storage backends (HDFS and SQLite) that can be freely mixed and matched so that a custom tool for a specific task can be easily constructed. BiDAl has a modular architecture so that it can be extended with other backends and analysis languages in the future. In this paper we present the design of BiDAl and describe our experience using it to analyze publicly-available traces from Google data clusters, with the goal of building a realistic model of a complex data center

Novel Push-Front Fibonacci Windows Model for Finding Emerging Patterns with Better Completeness and Accuracy

“…Our observation concluded that the time-complexity worst case of LWin is when all or almost all sup[i] and sub[i] are occupied, or N is almost reaching 2n, such as when N = 15. The volume sequence for N = 15 using LWin is {1, 1[1], 2 [2], 4 [4]}, and as shown, all sup[i] and sub[i], i > 0, are filled. Thus, when an EP evaluation is performed, the LWin algorithm must double-check all elements.…”

“…By contrast, FWin produces the lowest number of EPs because several elements are empty in some situations. For example, when N = 14, the volume sequences of PF-FWin, FWin, and LWin, respectively, are {1, 1, 1, 1, 2, 8}, {1, 0, 2, 3, 0, 8}, and {1, 1, 2 [2], 4 [4]}. While LWin even has two 1s in front of the array, FWin contains only one 1; its sup[1].vol is even empty, whereas it should provide the most accurate support.…”

“…By contrast, LWin found itemset{85099C} as an EP, but it could not be found by FWin. Itemset{85123A}, which is registered as a product called White hanging heart t-light holder, is found to be emerging by PF-FWin because its support growth ≥ 20% from sup [2] (75) to sup[0] (95). Using FWin and LWin, this itemset is not found to be emerging because the support growth threshold is not satisfied by sup[0] and the other elements.…”

“…With regard to accuracy, PF-FWin is able to provide supports with 100% accuracy in three elements, that is, from sup[0] to sup [2], since only one window is contained by such elements. LWin can still provide 100% accuracy in the current window (sup[0]) and the previous window (sup[1]); however, FWin can only provide the finest accuracy in the current window.…”

“…Two models are derived from TTWM. A model called the logarithmic tilted-time windows model (LWin) uses a sequence G = {1; 1[1]; 2 [2]; 4 [4]; 8 [8]; and so on}. Therefore, N windows can be accommodated in a (1 + 2 log(N))-elements array [10], while another model [11], called the Fibonacci windows model (FWin), uses a Fibonacci sequence F = {1, 1, 2, 3, 5, and so on} to organize the elements' capacities.…”