Design of the FutureGrid experiment management framework

Laszewski, Gregor von; Fox, Geoffrey; Wang, Fugang; Younge, Andrew J; Kulshrestha, Archit; Pike, Gregory G.; Smith, Warren; Vöckler, Jens; Figueiredo, Renato; Fortes, José A. B.; Keahey, Kate

doi:10.1109/gce.2010.5676126

Cited by 47 publications

(28 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The cloud resources were configured as a Condor pool using the Wrangler provisioning and configuration tool [14]. Wrangler, as mentioned above, allows the user to specify the number and type of resources to provision from a cloud provider and to specify what services (file systems, job schedulers, etc.)…”

Section: (B) Experiments On Academic Cloudsmentioning

confidence: 99%

The application of cloud computing to scientific workflows: a study of cost and performance

Berriman

Deelman

Juve

et al. 2013

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

The current model of transferring data from data centres to desktops for analysis will soon be rendered impractical by the accelerating growth in the volume of science datasets. Processing will instead often take place on high-performance servers co-located with data. Evaluations of how new technologies such as cloud computing would support such a new distributed computing model are urgently needed. Cloud computing is a new way of purchasing computing and storage resources on demand through virtualization technologies. We report here the results of investigations of the applicability of commercial cloud computing to scientific computing, with an emphasis on astronomy, including investigations of what types of applications can be run cheaply and efficiently on the cloud, and an example of an application well suited to the cloud: processing a large dataset to create a new science product.

show abstract

Section: (B) Experiments On Academic Cloudsmentioning

confidence: 99%

The application of cloud computing to scientific workflows: a study of cost and performance

Berriman

Deelman

Juve

et al. 2013

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

show abstract

“…Initially, ScaleMP vSMP was tested using a subset of the india supercomputer within the NSF FutureGrid project [15]. This consisted of 16 nodes removed from normal HPC usage, creating a 128 core single system with 384GB memory (320GB usable).…”

Section: A Echo At Iu Futuresystemsmentioning

confidence: 99%

Evaluation of SMP Shared Memory Machines for Use with In-Memory and OpenMP Big Data Applications

Younge

Reidy

Henschel

et al. 2016

2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

Self Cite

View full text Add to dashboard Cite

Abstract-While distributed memory systems have shaped the field of distributed systems for decades, the demand for many-core shared memory resources is increasing. Symmetric Multiprocessor Systems (SMPs) have become increasingly important recently among a wide array of disciplines, ranging from Bioinformatics to astrophysics, and beyond. With the increase in big data computing, the size and scope of traditional commodity server systems is often outpaced. While some big data applications can be mapped to distributed memory systems found through many cluster and cloud technologies today, this effort represents a large barrier of entry that some projects cannot cross. Shared memory SMP systems look to effectively and efficiently fill this niche within distributed systems by providing high throughput and performance with minimized development effort, as the computing environment often represents what many researchers are already familiar with.In this paper we look at the use of two common shared memory systems, the ScaleMP vSMP virtualized SMP deployment at Indiana University, and the SGI UV architecture deployed at University of Arizona. While both systems are notably different in their design, their potential impact on computing are remarkably similar. As such, we look to compare each system first under a set of OpenMP threaded benchmarks via the SPEC group, and follow up with our experience using each machine for Trinity de-novo assembly. We find both SMP systems are well suited to support various big data applications, with the newer vSMP deployment often slightly faster, however certain caveats and performance considerations are necessary when considering such SMP systems.

show abstract

“…We develop efficient data loading strategies that can accommodate fast loading of historical files as well as fast processing of streaming data from real-time tweets. We evaluate the performance of IndexedHBase on FutureGrid [20]. Our preliminary results show that, compared with Riak, IndexedHBase is significantly more efficient.…”

Section: Introductionmentioning

confidence: 99%

Supporting a Social Media Observatory with Customizable Index Structures: Architecture and Performance

Gao

Roth

McKelvey

et al. 2014

Cloud Computing for Data-Intensive Applications

View full text Add to dashboard Cite

Abstract. The intensive research activity in analysis of social media and microblogging data in recent years suggests the necessity and great potential of platforms that can efficiently store, query, analyze, and visualize social media data. To support these "social media observatories" effectively, a storage platform must satisfy special requirements for loading and storage of multi-terabyte datasets, as well as efficient evaluation of queries involving analysis of the text of millions of social updates. Traditional inverted indexing techniques do not meet such requirements. As a solution, we propose a general indexing framework, IndexedHBase, to build specially customized index structures for facilitating efficient queries on an HBase distributed data storage system. IndexedHBase is used to support a social media observatory that collects and analyzes data obtained through the Twitter streaming API. We develop a parallel query evaluation strategy that can explore the customized index structures efficiently, and test it on a set of typical social media data queries. We evaluate the performance of IndexedHBase on FutureGrid and compare it with Riak, a widely adopted commercial NoSQL database system. The results show that IndexedHBase provides a data loading speed that is six times faster than Riak and is significantly more efficient in evaluating queries involving large result sets.

show abstract

Design of the FutureGrid experiment management framework

Cited by 47 publications

References 6 publications

The application of cloud computing to scientific workflows: a study of cost and performance

The application of cloud computing to scientific workflows: a study of cost and performance

Evaluation of SMP Shared Memory Machines for Use with In-Memory and OpenMP Big Data Applications

Supporting a Social Media Observatory with Customizable Index Structures: Architecture and Performance

Contact Info

Product

Resources

About