Processing Cassandra Datasets with Hadoop-Streaming Based Approaches

Abstract

The progressive transition in the nature of both scientific and industrial datasets has been the driving force behind the development and research interests in the NoSQL model. Loosely structured data poses a challenge to traditional data store systems, and when working with the NoSQL model, these systems are often considered impractical and costly. As the quantity and quality of unstructured data grows, so does the demand for a processing pipeline that is capable of seamlessly combining the NoSQL storage model and a "Big Data" processing platform such as MapReduce. Although MapReduce is the paradigm of choice for data-intensive computing, Java-based frameworks such as Hadoop require users to write MapReduce code in Java while Hadoop Streaming module allows users to define non-Java executables as map and reduce operations. When confronted with legacy C/C++ applications and other non-Java executables, there arises a further need to allow NoSQL data stores access to the features of Hadoop Streaming. We present approaches in solving the challenge of integrating NoSQL data stores with MapReduce under non-Java application scenarios, along with advantages and disadvantages of each approach. We compare Hadoop Streaming alongside our own streaming framework, MARISSA, to show performance implications of coupling NoSQL data stores like Cassandra with MapReduce frameworks that normally rely on file-system based data stores. Our experiments also include Hadoop-C*, which is a setup where a Hadoop cluster is co-located with a Cassandra cluster in order to process data using Hadoop with non-java executables.

Error Concealment for Cloud–Based and Scalable Video Coding of HD Videos

“…Another similar approach was presented in [6] and focused on reducing network overhead. To deal with the storage and processing of large media databases, MapReduce was applied along with a Java-based framework for Hadoop streaming in [8]. Cloud platform was used to efficiently deal with video processing in [14], [15].…”

“…Cloud platform was used to efficiently deal with video processing in [14], [15]. All of the approaches presented in [5], [6], [8], [14], [15] focused on the cloud side, i.e., the server side and ignored the packet loss problems.…”

“…Media cloud platforms are becoming popular to provide video streaming services. Recently, there have been little research exploring how a cloud platform can be utilized efficiently for video streaming services in [5]- [8]. To meet end-user's demands for real-time services, a VSP can rent cloud resources, including storages and virtual CPUs, in order to avoid the investment on hardware and its maintenance.…”

Healthcare Data Management and Analytics using Big Data Tools

“…Once placed in HDFS, the data can be used by Hadoop applications. Flume: Apache Flume is a extremely reliable service for accurate information collection and transition from autonomous computers to HDFS [11]. A number of flume agents that can cross a sequence of computers and places are often involved in information transportation.…”

Reimaging Research Methodology as Data Science

“…Big Data is also associated with the new forms of digital data generated from social media and learning management systems offer researchers enormous opportunities for extracting meaning (Dede et al, 2016;Doorn, 2014; Lazer, Kennedy, King, and Vespignani, 2014) [33][34][35].…”