The big data ecosystem at LinkedIn

Sumbaly, Roshan; Kreps, Jay; Shah, Sam

doi:10.1145/2463676.2463707

Cited by 115 publications

(57 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They lack parallelism model support for processing but are often combined with other platforms to deliver full real-time data analytics and processing [28,29]. We found that in our case, both Kafka and Flume can be used to load the CDRs to the processing nodes but as they are optimized for streams, this approach may introduce overhead for splitting larger sizes of batched CDRs to smaller chunks.…”

Section: Related Workmentioning

confidence: 98%

High Performance CDR Processing with MapReduce

Agung

Kistijantoro

2016

J. ICT Res. Appl.

View full text Add to dashboard Cite

Abstract.A call detail record (CDR) is a data record produced by telecommunication equipment consisting of call detail transaction logs. It contains valuable information for many purposes in several domains, such as billing, fraud detection and analytical purposes. However, in the real world these needs face a big data challenge. Billions of CDRs are generated every day and the processing systems are expected to deliver results in a timely manner. The capacity of our current production system is not enough to meet these needs. Therefore a better performing system based on MapReduce and running on Hadoop cluster was designed and implemented. This paper presents an analysis of the previous system and the design and implementation of the new system, called MS2. In this paper also empirical evidence is provided to demonstrate the efficiency and linearity of MS2. Tests have shown that MS2 reduces overhead by 44% and speeds up performance nearly twice compared to the previous system. From benchmarking with several related technologies in large-scale data processing, MS2 was also shown to perform better in the case of CDR batch processing. When it runs on a cluster consisting of eight CPU cores and two conventional disks, MS2 is able to process 67,000 CDRs/second.

show abstract

Section: Related Workmentioning

confidence: 98%

High Performance CDR Processing with MapReduce

Agung

Kistijantoro

2016

J. ICT Res. Appl.

View full text Add to dashboard Cite

show abstract

“…Consequently, distributed and streaming applications of machine learning algorithms 14 should be explored to effectively model the large corpus of online reviews which exist in the real world [38]. Mahout has been used for large-scale recommendation systems [39], which would be useful to apply to review spam detection, as reviewers may be related to each other on different review websites. MLlib and SAMOA can perform large-scale online learning, where machine learning models are trained and tuned as new data flows in.…”

Section: Comparative Analysis and Suggestionsmentioning

confidence: 99%

Survey of review spam detection using machine learning techniques

Crawford

Khoshgoftaar

Prusa

et al. 2015

Journal of Big Data

369

176

View full text Add to dashboard Cite

Online reviews are often the primary factor in a customer's decision to purchase a product or service, and are a valuable source of information that can be used to determine public opinion on these products or services. Because of their impact, manufacturers and retailers are highly concerned with customer feedback and reviews. Reliance on online reviews gives rise to the potential concern that wrongdoers may create false reviews to artificially promote or devalue products and services. This practice is known as Opinion (Review) Spam, where spammers manipulate and poison reviews (i.e., making fake, untruthful, or deceptive reviews) for profit or gain. Since not all online reviews are truthful and trustworthy, it is important to develop techniques for detecting review spam. By extracting meaningful features from the text using Natural Language Processing (NLP), it is possible to conduct review spam detection using various machine learning techniques. Additionally, reviewer information, apart from the text itself, can be used to aid in this process. In this paper, we survey the prominent machine learning techniques that have been proposed to solve the problem of review spam detection and the performance of different approaches for classification and detection of review spam. The majority of current research has focused on supervised learning methods, which require labeled data, a scarcity when it comes to online review spam. Research on methods for Big Data are of interest, since there are millions of online reviews, with many more being generated daily. To date, we have not found any papers that study the effects of Big Data analytics for review spam detection. The primary goal of this paper is to provide a strong and comprehensive comparative study of current research on detecting review spam using various machine learning techniques and to devise methodology for conducting further investigation.

show abstract

“…There has also been a huge interest and opportunity of big data in the health industry. Facebook and LinkedIn collect from both traditional database and streaming data from users whereas Twitter mostly deals with streaming data [26,27,28,29,32,33]. The collected data are then handled on a batch or streaming processing with each own defined processing functionalities.…”

Section: A Big Data Use Casementioning

confidence: 99%

A reference architecture for big data systems

Sang

Vrieze

2016

2016 10th International Conference on Software, Knowledge, Information Management &Amp; Applications (SKIMA)

View full text Add to dashboard Cite

Over dozens of years, applying new IT technologies into organizations has always been a big concern for business. Big data certainly is a new concept exciting business. To be able to access more data and empower to analysis big data requires new big data platforms. However, there still remains limited reference architecture for big data systems. In this paper, based on existing reference architecture of big data systems, we propose new high level abstract reference architecture and related reference architecture notations, that better express the overall architecture. The new reference architecture is verified using one existing case and an additional new use case.

show abstract

The big data ecosystem at LinkedIn

Cited by 115 publications

References 24 publications

High Performance CDR Processing with MapReduce

High Performance CDR Processing with MapReduce

Survey of review spam detection using machine learning techniques

A reference architecture for big data systems

Contact Info

Product

Resources

About