Proposal of Lambda Architecture Adoption for Real Time Predictive Maintenance

Yamato, Yoji; Kumazaki, Hiroki; Fukumoto, Yoshihiro

doi:10.1109/candar.2016.0130

Cited by 29 publications

(13 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is also a promising method that reduces significantly the processing time and the resource requirements. Moreover, Yamato et al [32] used a lambda architecture to analyze data from IoT sensors. It is a data analytics framework that uses incremental learning techniques to identify anomalies in real time.…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

The Next Generation Cognitive Security Operations Center: Adaptive Analytic Lambda Architecture for Efficient Defense against Adversarial Attacks

Demertzis

Tziritas

Kikiras

et al. 2019

BDCC

View full text Add to dashboard Cite

A Security Operations Center (SOC) is a central technical level unit responsible for monitoring, analyzing, assessing, and defending an organization’s security posture on an ongoing basis. The SOC staff works closely with incident response teams, security analysts, network engineers and organization managers using sophisticated data processing technologies such as security analytics, threat intelligence, and asset criticality to ensure security issues are detected, analyzed and finally addressed quickly. Those techniques are part of a reactive security strategy because they rely on the human factor, experience and the judgment of security experts, using supplementary technology to evaluate the risk impact and minimize the attack surface. This study suggests an active security strategy that adopts a vigorous method including ingenuity, data analysis, processing and decision-making support to face various cyber hazards. Specifically, the paper introduces a novel intelligence driven cognitive computing SOC that is based exclusively on progressive fully automatic procedures. The proposed λ-Architecture Network Flow Forensics Framework (λ-ΝF3) is an efficient cybersecurity defense framework against adversarial attacks. It implements the Lambda machine learning architecture that can analyze a mixture of batch and streaming data, using two accurate novel computational intelligence algorithms. Specifically, it uses an Extreme Learning Machine neural network with Gaussian Radial Basis Function kernel (ELM/GRBFk) for the batch data analysis and a Self-Adjusting Memory k-Nearest Neighbors classifier (SAM/k-NN) to examine patterns from real-time streams. It is a forensics tool for big data that can enhance the automate defense strategies of SOCs to effectively respond to the threats their environments face.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

“…In addition, a speed layer (hot path) analyzes unbounded streams of data in real time. The hot path is planned for low latency, at the expense of accuracy [32]. Figure 1 is a depiction of the lambda architecture.…”

Section: The Proposed Approachmentioning

confidence: 99%

The Next Generation Cognitive Security Operations Center: Adaptive Analytic Lambda Architecture for Efficient Defense against Adversarial Attacks

Demertzis

Tziritas

Kikiras

et al. 2019

BDCC

View full text Add to dashboard Cite

show abstract

“…In their work, Yamato et al [2] illustrated a model for real-time predictive maintenance on an IoT deployment for NTT DOCOMO. The model analyzes the sensor stream data for anomalies at the speed layer, which requires prompt actions.…”

Section: Background and Related Workmentioning

confidence: 99%

Multi-Agent Big-Data Lambda Architecture Model for E-Commerce Analytics

Pal

Atkinson

2018

Data

View full text Add to dashboard Cite

We study big-data hybrid-data-processing lambda architecture, which consolidates low-latency real-time frameworks with high-throughput Hadoop-batch frameworks over a massively distributed setup. In particular, real-time and batch-processing engines act as autonomous multi-agent systems in collaboration. We propose a Multi-Agent Lambda Architecture (MALA) for e-commerce data analytics. We address the high-latency problem of Hadoop MapReduce jobs by simultaneous processing at the speed layer to the requests which require a quick turnaround time. At the same time, the batch layer in parallel provides comprehensive coverage of data by intelligent blending of stream and historical data through the weighted voting method. The cold-start problem of streaming services is addressed through the initial offset from historical batch data. Challenges of high-velocity data ingestion is resolved with distributed message queues. A proposed multi-agent decision-maker component is placed at the MALA stack as the gateway of the data pipeline. We prove efficiency of our batch model by implementing an array of features for an e-commerce site. The novelty of the model and its key significance is a scheme for multi-agent interaction between batch and real-time agents to produce deeper insights at low latency and at significantly lower costs. Hence, the proposed system is highly appealing for applications involving big data and caters to high-velocity streaming ingestion and a massive data pool.

show abstract

“…Users also can coordinate outer systems using service coordination technologies such as [61][62][63][64][65][66]. We also consider to use GPU/ FPGA servers to IoT applications [67][68][69][70][71][72][73][74].…”

Section: Related Workmentioning

confidence: 99%

Server Selection, Configuration and Reconfiguration Technology for IaaS Cloud with Multiple Server Types

Yamato

2017

J Netw Syst Manage

Self Cite

View full text Add to dashboard Cite

We propose a server selection, configuration, reconfiguration and automatic performance verification technology to meet user functional and performance requirements on various types of cloud compute servers. Various servers mean there are not only virtual machines on normal CPU servers but also container or baremetal servers on strong graphic processing unit (GPU) servers or field programmable gate arrays (FPGAs) with a configuration that accelerates specified computation. Early cloud systems are composed of many PC-like servers, and virtual machines on these severs use distributed processing technology to achieve high computational performance. However, recent cloud systems change to make the best use of advances in hardware power. It is well known that baremetal and container performances are better than virtual machines performances. And dedicated processing servers, such as strong GPU servers for graphics processing, and FPGA servers for specified computation, have increased. Our objective for this study was to enable cloud providers to provision compute resources on appropriate hardware based on user requirements, so that users can benefit from high performance of their applications easily. Our proposed technology select appropriate servers for user compute resources from various types of hardware, such as GPUs and FPGAs, or set appropriate configurations or reconfigurations of FPGAs to use hardware power. Furthermore, our technology automatically verifies the performances of provisioned systems. We measured provisioning and automatic performance verification times to show the effectiveness of our technology.

show abstract

Proposal of Lambda Architecture Adoption for Real Time Predictive Maintenance

Cited by 29 publications

References 8 publications

The Next Generation Cognitive Security Operations Center: Adaptive Analytic Lambda Architecture for Efficient Defense against Adversarial Attacks

The Next Generation Cognitive Security Operations Center: Adaptive Analytic Lambda Architecture for Efficient Defense against Adversarial Attacks

Multi-Agent Big-Data Lambda Architecture Model for E-Commerce Analytics

Server Selection, Configuration and Reconfiguration Technology for IaaS Cloud with Multiple Server Types

Contact Info

Product

Resources

About