A convergence of key‐value storage systems from clouds to supercomputers

Li, Tonglin; Zhou, Xiaobing; Wang, Ke; Zhao, Dongfang; Sadooghi, Iman; Zhang, Zhao; Raicu, Ioan

doi:10.1002/cpe.3614

Cited by 24 publications

(6 citation statements)

References 50 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The goal of knowledge map construction is to convert a knowledge database into a corresponding knowledge map to find knowledge resources and increase knowledge sharing [12,18]. A knowledge map consists of vertices and edges.…”

Section: Methodsmentioning

confidence: 99%

“…Research on the knowledge map has extended from the original connotation and functions to construction techniques. Although many scholars have studied the meaning and functions of knowledge maps, no uniform construction method of knowledge maps has been reported because knowledge maps have different roles and representations in different application situations [12]. Some researchers have explored manual construction of knowledge maps thoroughly.…”

Section: Related Workmentioning

confidence: 99%

“…development process and Knowledge has become increasingly critical to improve the efficiency and quality of product development [5,7,10,12,20]. Many organizations deploy knowledge management systems to gain a competitive edge and accumulate a substantial amount of knowledge.…”

mentioning

confidence: 99%

See 2 more Smart Citations

A novel method for adaptive knowledge map construction in the aircraft development

Zhao

2015

Multimed Tools Appl

View full text Add to dashboard Cite

Aircraft is a typical transportation facility and its development need to refer to the existing knowledge. With the rapid increase of knowledge, a knowledge map may deliver excess knowledge to users that they cannot manage at once, thereby causing the problem of knowledge overload. Hence, a novel method for adaptive knowledge map construction was proposed to solve this problem. First, the knowledge was semantically annotated and stored with the domain ontology and a knowledge model that integrates context. Then, user requirement was described by the context of product design, and knowledge nodes that met users' requirement could be extracted from the knowledge retrieval technology on the basis of context similarity. Finally, the connection between knowledge nodes was constructed with a composite connection model, and the knowledge map was visualized using a hierarchical approach. To verify the effectiveness of the proposed method, the constructed knowledge map was applied in an airplane wing design to assist users in browsing the knowledge base. Results indicate that the proposed method can change the displayed contents according to user requirement and identify the displayed knowledge nodes at a highly acceptable level, the constructed knowledge map can guide users efficiently, and the knowledge overload can be reduced significantly.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A novel method for adaptive knowledge map construction in the aircraft development

Zhao

2015

Multimed Tools Appl

View full text Add to dashboard Cite

show abstract

“…Since these systems follow the client-server-architecture, the user would have to setup the DHT as an additional software component which runs separately. While systems like D1HT (Monnerat and Amorim, 2015) use their own communication framework, there exist several MPI-based designs (see for example Tsukamoto and Nakashima, 2010;Wozniak et al, 2010;Li et al, 2016;Maynard, 2011).…”

Section: Distributed Hash Tables: Related Workmentioning

confidence: 99%

<tt>POET</tt> (v0.1): Speedup of Many-Cores Parallel Reactive Transport Simulations with Fast DHT-Lookups

Lucia

Kühn

Lindemann

et al. 2021

Preprint

View full text Add to dashboard Cite

Abstract. Coupled reactive transport simulations are extremely demanding in terms of required computational power, which hampers their application and leads to coarsened and oversimplified domains. The chemical sub-process represents the major bottleneck: its acceleration is an urgent challenge which gathers increasing interdisciplinary interest along with pressing requirements for subsurface utilization such as spent nuclear fuel storage, geothermal energy and CO2 storage. In this context we 5 developed POET (POtsdam rEactive Transport), a research parallel reactive transport simulator integrating algorithmic improvements which decisively speedup coupled simulations. In particular, POET is designed with a master/worker architecture, which ensures computational efficiency on both multicore and cluster compute environments. POET does not rely on contiguous grid partitions for the parallelization of chemistry, but forms work packages composed of grid cells distant from each other. Such scattering prevents particularly expensive geochemical simulations, usually concentrated in the vicinity of a reactive front, from generating load imbalance between the available CPUs, as it is often the case with classical partitions. Furthermore, POET leverages an original implementation of Distributed Hash Table (DHT) mechanism to cache the results of geochemical simulations for further reuse in subsequent time-steps during the coupled simulation. The caching is hence particularly advantageous for initially chemically homogeneous simulations and for smooth reaction fronts. We tune the rounding employed in the DHT on a 2D benchmark to validate the caching approach, and we evaluate the performance gain of POET's master/worker architecture and the DHT speedup on a 3D benchmark comprising around 650 k grid elements. The runtime for 200 coupling iterations, corresponding to 960 simulation days, reduced from about 24 h on 11 workers to 29 minutes on 719 workers. Activating the DHT reduces the runtime further to 2 h and 8 minutes respectively. Only with this kind of reduced hardware requirements and computational costs it is possible to realistically perform the large scale, long-term complex reactive transport simulations, as well as performing the uncertainty analyses required by pressing societal challenges connected with subsurface utilization.

show abstract

“…A ready task was put in either queue based on the size and location of the required input data. A distributed key‐value store (KVS) (i.e., ZHT ) was applied as a metadata service to keep task metadata, including data dependency and locality information, for all the tasks. We implemented the technique in MATRIX , an MTC task execution framework, and evaluated up to 200 cores.…”

Section: Introductionmentioning

confidence: 99%

Load‐balanced and locality‐aware scheduling for data‐intensive workloads at extreme scales

Wang

Qiao

Sadooghi

et al. 2015

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Summary Data‐driven programming models such as many‐task computing (MTC) have been prevalent for running data‐intensive scientific applications. MTC applies over‐decomposition to enable distributed scheduling. To achieve extreme scalability, MTC proposes a fully distributed task scheduling architecture that employs as many schedulers as the compute nodes to make scheduling decisions. Achieving distributed load balancing and best exploiting data locality are two important goals for the best performance of distributed scheduling of data‐intensive applications. Our previous research proposed a data‐aware work‐stealing technique to optimize both load balancing and data locality by using both dedicated and shared task ready queues in each scheduler. Tasks were organized in queues based on the input data size and location. Distributed key‐value store was applied to manage task metadata. We implemented the technique in MATRIX, a distributed MTC task execution framework. In this work, we devise an analytical suboptimal upper bound of the proposed technique, compare MATRIX with other scheduling systems, and explore the scalability of the technique at extreme scales. Results show that the technique is not only scalable but can achieve performance within 15% of the suboptimal solution. Copyright © 2015 John Wiley & Sons, Ltd.

show abstract

A convergence of key‐value storage systems from clouds to supercomputers

Cited by 24 publications

References 50 publications

A novel method for adaptive knowledge map construction in the aircraft development

A novel method for adaptive knowledge map construction in the aircraft development

<tt>POET</tt> (v0.1): Speedup of Many-Cores Parallel Reactive Transport Simulations with Fast DHT-Lookups

Load‐balanced and locality‐aware scheduling for data‐intensive workloads at extreme scales

Contact Info

Product

Resources

About

A convergence of key‐value storage systems from clouds to supercomputers

Cited by 24 publications

References 50 publications

A novel method for adaptive knowledge map construction in the aircraft development

A novel method for adaptive knowledge map construction in the aircraft development

&lt;tt&gt;POET&lt;/tt&gt; (v0.1): Speedup of Many-Cores Parallel Reactive Transport Simulations with Fast DHT-Lookups

Load‐balanced and locality‐aware scheduling for data‐intensive workloads at extreme scales

Contact Info

Product

Resources

About

<tt>POET</tt> (v0.1): Speedup of Many-Cores Parallel Reactive Transport Simulations with Fast DHT-Lookups