Galaxy + Hadoop: Toward a Collaborative and Scalable Image Processing Toolbox in Cloud

Chen, Jinjun; Bednarz, Tomasz; Szul, Piotr; Wang, Dadong; Arzhaeva, Yulia; Burdett, Neil; Khassapov, Alex; Zic, John; Nepal, Surya; Gurevey, Tim; Taylor, John A.

doi:10.1007/978-3-319-06859-6_30

Cited by 8 publications

(3 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hadoop and MapReduce have become the most frequently used image processing platforms [48]- [51]. These parallel processing techniques favor the use of computing resources on the cloud for processing multimedia data, offering very competitive proposals for massive parallel processing [52], [53]. Additionally, the cloud computing paradigm is emerging as a solution to supply specific computing resources for applications with massive computing needs.…”

Section: B Multimedia Architecturesmentioning

confidence: 99%

Distributed Architectures for Intensive Urban Computing: A Case Study on Smart Lighting for Sustainable Cities

et al. 2019

View full text Add to dashboard Cite

New information and communication technologies have contributed to the development of the smart city concept. On a physical level, this paradigm is characterized by deploying a substantial number of different devices that can sense their surroundings and generate a large amount of data. The most typical case is image and video acquisition sensors. Recently, these types of sensors are found in abundance in urban spaces and are responsible for producing a large volume of multimedia data. The advanced computer vision methods for this type of multimedia information means that many aspects can be dynamically monitored, which can help implement value-added applications in the city. However, obtaining more elaborate semantic information from these data poses significant challenges related to a large amount of data generated and the processing capabilities required. This paper aims to address these issues by using a combination of cloud computing technologies and mobile computing techniques to design a three-layer distributed architecture for intensive urban computing. The approach consists of distributing the processing tasks among a city's multimedia acquisition devices, a middle computing layer, known as a cloudlet, and a cloud-computing infrastructure. As a result, each part of the architecture can now focus on a small number of tasks for which they are specially designed, and data transmission communication needs are significantly reduced. To this end, the cloud server can hold and centralize the multimedia analysis of the processed results from the lower layers. Finally, a case study on smart lighting is described to illustrate the benefits of using the proposed model in smart city environments. INDEX TERMS Mobile cloud computing, data processing, distributed architectures, smart city, urban computing.

show abstract

Section: B Multimedia Architecturesmentioning

confidence: 99%

Distributed Architectures for Intensive Urban Computing: A Case Study on Smart Lighting for Sustainable Cities

et al. 2019

View full text Add to dashboard Cite

show abstract

“…It can be seen that the Hadoop framework is a potential execution environment for many cases of remote sensing image analysis using commodity hardware. Studies showed that a complete image management and processing framework can be built on the top of Hadoop platform using standardized technology and open-source software [9,27,40].…”

Section: Related Studies On Distributed Image Processingmentioning

confidence: 99%

A Case Study of Advancing Remote Sensing Image Analysis

Giachetta¹,

Fekete²

2015

Acta Cybern

View full text Add to dashboard Cite

Big data and cloud computing are two phenomena, which have gained significant reputation over the last few years. In computer science the approach shifted towards distributed architectures and high performance computing. In case of geographical information systems (GIS) and remote sensing image analysis, the new paradigms have already been successfully applied to several problems, and systems have been developed to support processing of geographical and remote sensing data in the cloud. However, due to different circumstances many previous workflows have to be reconsidered and redesigned.Our goal is to show a way how the existing approaches to remote sensing image analysis can be advanced to take advantages of these new paradigms. The task aiming in shifting the algorithms shall require a moderate effort and must avoid the complete redesign and reimplementation of the existing approaches. We present the whole journey as a case study using an existing industrial workflow for demonstration. Nevertheless, we define the rules of thumb, which can come in hand when shifting any existing GIS workflows.Our case study is the workflow of waterlogging and flood detection, which is an operative task at the Institute of Geodesy, Cartography and Remote Sensing (FÖMI). This task in currently operational using a semi-automatic single machine approach involving multiple software. The workflow is neither efficient nor scalable, thus it is not applicable in emergency situations where quick response is required. We present an approach utilizing distributed computing, which enables the automated execution of this task on large input data with much better response time. The approach is based on the well-known MapReduce paradigm, its open-source implementation, the Apache Hadoop framework and the AEGIS geospatial toolkit. This enables the replacement of multiple software to a single, generic framework. Results show that significant performance benefits can be achieved at the expense of minor accuracy loss.Keywords: remote sensing image analysis, object based image classification, * This research is founded by the FP7 project IQmulus (FP7-ICT-2011-318787) a high volume fusion and analysis platform for geospatial point clouds, coverages and volumetric data set.

show abstract

“…Second, several approaches have followed the path of general machine learning literature and seek to implement algorithms specifically designed to take advantage of big data architecture [6], [11], [32], exploit the MapReduce framework to sift through datasets [25], or use distributed file systems [30], [36]. While such approaches have been effective for genetics studies [9], [36], they have not yet proven effective within current medical image computing workflows.…”

Section: Introductionmentioning

confidence: 99%

Cloud Engineering Principles and Technology Enablers for Medical Image Processing-as-a-Service

Bao

Plassard

Landman

et al. 2017

2017 IEEE International Conference on Cloud Engineering (IC2E)

View full text Add to dashboard Cite

Traditional in-house, laboratory-based medical imaging studies use hierarchical data structures (e.g., NFS file stores) or databases (e.g., COINS, XNAT) for storage and retrieval. The resulting performance from these approaches is, however, impeded by standard network switches since they can saturate network bandwidth during transfer from storage to processing nodes for even moderate-sized studies. To that end, a cloud-based “medical image processing-as-a-service” offers promise in utilizing the ecosystem of Apache Hadoop, which is a flexible framework providing distributed, scalable, fault tolerant storage and parallel computational modules, and HBase, which is a NoSQL database built atop Hadoop’s distributed file system. Despite this promise, HBase’s load distribution strategy of region split and merge is detrimental to the hierarchical organization of imaging data (e.g., project, subject, session, scan, slice). This paper makes two contributions to address these concerns by describing key cloud engineering principles and technology enhancements we made to the Apache Hadoop ecosystem for medical imaging applications. First, we propose a row-key design for HBase, which is a necessary step that is driven by the hierarchical organization of imaging data. Second, we propose a novel data allocation policy within HBase to strongly enforce collocation of hierarchically related imaging data. The proposed enhancements accelerate data processing by minimizing network usage and localizing processing to machines where the data already exist. Moreover, our approach is amenable to the traditional scan, subject, and project-level analysis procedures, and is compatible with standard command line/scriptable image processing software. Experimental results for an illustrative sample of imaging data reveals that our new HBase policy results in a three-fold time improvement in conversion of classic DICOM to NiFTI file formats when compared with the default HBase region split policy, and nearly a six-fold improvement over a commonly available network file system (NFS) approach even for relatively small file sets. Moreover, file access latency is lower than network attached storage.

show abstract

Galaxy + Hadoop: Toward a Collaborative and Scalable Image Processing Toolbox in Cloud

Cited by 8 publications

References 14 publications

Distributed Architectures for Intensive Urban Computing: A Case Study on Smart Lighting for Sustainable Cities

Distributed Architectures for Intensive Urban Computing: A Case Study on Smart Lighting for Sustainable Cities

A Case Study of Advancing Remote Sensing Image Analysis

Cloud Engineering Principles and Technology Enablers for Medical Image Processing-as-a-Service

Contact Info

Product

Resources

About