<b>rEMM</b>: Extensible Markov Model for Data Stream Clustering in<i>R</i>

Hahsler, Michael; Dunham, Margaret H.

doi:10.18637/jss.v035.i05

Cited by 13 publications

(8 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the second case, a typical example is the Temporal Structure Learning for Clustering Massive Data Stream in Real Time (TRACDS ) algorithm [Hahsler and Dunham 2011]. It is essentially a generalization of the Extensible Markov Model (EMM) algorithm [Dunham et al 2004;Hahsler and Dunham 2010] for data stream scenarios. In TRACDS, each cluster (or micro-cluster) is represented by a state of a Markov Chain (MC) [Markov 1971;Bhat and Miller 2002], and the transitions represent the relationship between clusters.…”

Section: Time-aware Clusteringmentioning

confidence: 99%

Data stream clustering

et al. 2013

View full text Add to dashboard Cite

Data stream mining is an active research area that has recently emerged to discover knowledge from large amounts of continuously generated data. In this context, several data stream clustering algorithms have been proposed to perform unsupervised learning. Nevertheless, data stream clustering imposes several challenges to be addressed, such as dealing with nonstationary, unbounded data that arrive in an online fashion. The intrinsic nature of stream data requires the development of algorithms capable of performing fast and incremental processing of data objects, suitably addressing time and memory limitations. In this article, we present a survey of data stream clustering algorithms, providing a thorough discussion of the main design components of state-of-the-art algorithms. In addition, this work addresses the temporal aspects involved in data stream clustering, and presents an overview of the usually employed experimental methodologies. A number of references are provided that describe applications of data stream clustering in different domains, such as network intrusion detection, sensor networks, and stock market analysis. Information regarding software packages and data repositories are also available for helping researchers and practitioners. Finally, some important issues and open questions that can be subject of future research are discussed.

show abstract

Section: Time-aware Clusteringmentioning

confidence: 99%

Data stream clustering

et al. 2013

View full text Add to dashboard Cite

show abstract

“…This count is incremented by one when the cluster is being assigned with a new observation. Cluster fading is represented by applying fading weights on the clusters observation counts (in this case we call them weighted observations counts) like in [17].…”

Section: A Implementation Requirementsmentioning

confidence: 99%

“…In our study, we use the GNU R implementation of the data stream clustering approach from [17]. It is a simple implementation of the threshold nearest neighbor algorithm.…”

Section: Data Stream Clustering Implementationmentioning

confidence: 99%

See 1 more Smart Citation

Data Stream Clustering for Online Anomaly Detection in Cloud Applications

Sauvanaud

Silvestre²,

Kaâniche³

et al. 2015

2015 11th European Dependable Computing Conference (EDCC)

View full text Add to dashboard Cite

This paper introduces a new approach for the online detection of performance anomalies in cloud virtual machines (VMs). It is designed for cloud infrastructure providers to detect during runtime unknown anomalies that may still be observed in complex modern systems hosted on VMs. The approach is drawn on data stream clustering of per-VM monitoring data and detects at a fine granularity where anomalies occur. It is design to be independent of the types of applications deployed over VMs. Moreover it deals with relentless changes in systems normal behaviors during runtime. The parallel analyses of each VM makes this approach scalable to a large number of VMs composing an application. The approach consists of two online steps: 1) the incremental update of sets of clusters by means of data stream clustering, and 2) the computation of two attributes characterizing the global clusters evolution. We validate our approach over a VMware vSphere testbed. It hosts a typical cloud application, MongoDB, that we study in normal behavior contexts and in presence of anomalies.

show abstract

“…This package has methods to quickly load a large set of sequence files, that can be in FASTA format with Greengenes [ 22 ] annotations, into a relational database and can be used to easily filter sequences belonging to any taxonomic rank. This package is built on top of a number of other packages including Biostrings [ 29 ] for handling sequences, and the data stream clustering package rEMM [ 30 , 31 ]. It provides a complete interface for managing sequences, creating word frequencies distributions (NSVs) and creating and analyzing GenModels.…”

Section: Quasi-alignment Via Position-sensitive P mentioning

confidence: 99%

Fast discovery and visualization of conserved regions in DNA sequences using quasi-alignment

Nagar

Hahsler

2013

BMC Bioinformatics

Self Cite

View full text Add to dashboard Cite

BackgroundNext Generation Sequencing techniques are producing enormous amounts of biological sequence data and analysis becomes a major computational problem. Currently, most analysis, especially the identification of conserved regions, relies heavily on Multiple Sequence Alignment and its various heuristics such as progressive alignment, whose run time grows with the square of the number and the length of the aligned sequences and requires significant computational resources. In this work, we present a method to efficiently discover regions of high similarity across multiple sequences without performing expensive sequence alignment. The method is based on approximating edit distance between segments of sequences using p-mer frequency counts. Then, efficient high-throughput data stream clustering is used to group highly similar segments into so called quasi-alignments. Quasi-alignments have numerous applications such as identifying species and their taxonomic class from sequences, comparing sequences for similarities, and, as in this paper, discovering conserved regions across related sequences.ResultsIn this paper, we show that quasi-alignments can be used to discover highly similar segments across multiple sequences from related or different genomes efficiently and accurately. Experiments on a large number of unaligned 16S rRNA sequences obtained from the Greengenes database show that the method is able to identify conserved regions which agree with known hypervariable regions in 16S rRNA. Furthermore, the experiments show that the proposed method scales well for large data sets with a run time that grows only linearly with the number and length of sequences, whereas for existing multiple sequence alignment heuristics the run time grows super-linearly.ConclusionQuasi-alignment-based algorithms can detect highly similar regions and conserved areas across multiple sequences. Since the run time is linear and the sequences are converted into a compact clustering model, we are able to identify conserved regions fast or even interactively using a standard PC. Our method has many potential applications such as finding characteristic signature sequences for families of organisms and studying conserved and variable regions in, for example, 16S rRNA.

show abstract

rEMM: Extensible Markov Model for Data Stream Clustering inR

Cited by 13 publications

References 35 publications

Data stream clustering

Data stream clustering

Data Stream Clustering for Online Anomaly Detection in Cloud Applications

Fast discovery and visualization of conserved regions in DNA sequences using quasi-alignment

Contact Info

Product

Resources

About