A Consistency Framework for Iteration Operations in Concurrent Data Structures

Nikolakopoulos, Yiannis; Gidenstam, Anders; Papatriantafilou, Marina; Tsigas, Philippas

doi:10.1109/ipdps.2015.84

Cited by 12 publications

(6 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We propose the extensible generic algorithmic framework Leashed-SGD for asynchronous lock-free parallel SGD, together with ParameterVector, a data type providing an abstraction of common operations on high-dimensional model parameters in ANN training, facilitating modular further exploration of aspects of parallelism and consistency, connecting to and extending work in the literature on bulk operations on container data structures [34].…”

Section: Discussionmentioning

confidence: 99%

“…section II). Hence studying shared memory data-parallel SGD implementations with synchronization in focus, is to study implications of the properties of the algorithmic implementations of the parameter vector seen as shared object, connecting to and extending work in the literature on bulk operations on container data structures [34].…”

Section: The Leashed-sgd Frameworkmentioning

confidence: 99%

See 1 more Smart Citation

Consistent Lock-free Parallel Stochastic Gradient Descent for Fast and Stable Convergence

Bäckström¹,

Walulya²,

Papatriantafilou³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Stochastic gradient descent (SGD) is an essential element in Machine Learning (ML) algorithms. Asynchronous parallel shared-memory SGD (AsyncSGD), including synchronization-free algorithms, e.g. HOGWILD!, have received interest in certain contexts, due to reduced overhead compared to synchronous parallelization. Despite that they induce staleness and inconsistency, they have shown speedup for problems satisfying smooth, strongly convex targets, and gradient sparsity. Recent works take important steps towards understanding the potential of parallel SGD for problems not conforming to these strong assumptions, in particular for deep learning (DL). There is however a gap in current literature in understanding when AsyncSGD algorithms are useful in practice, and in particular how mechanisms for synchronization and consistency play a role.We contribute with answering questions in this gap by studying a spectrum of parallel algorithmic implementations of AsyncSGD, aiming to understand how shared-data synchronization influences the convergence properties in fundamental DL applications. We focus on the impact of consistency-preserving non-blocking synchronization in SGD convergence, and in sensitivity to hyperparameter tuning. We propose Leashed-SGD, an extensible algorithmic framework of consistency-preserving implementations of AsyncSGD, employing lock-free synchronization, effectively balancing throughput and latency. Leashed-SGD features a natural contention-regulating mechanism, as well as dynamic memory management, allocating space only when needed. We argue analytically about the dynamics of the algorithms, memory consumption, the threads' progress over time, and the expected contention. The analysis further shows the contention-regulating mechanism that Leashed-SGD enables.We provide a comprehensive empirical evaluation, validating the analytical claims, benchmarking the proposed Leashed-SGD framework, and comparing to baselines for two prominent deep learning (DL) applications: multilayer perceptrons (MLP) and convolutional neural networks (CNN). We observe the crucial impact of contention, staleness and consistency and show how, thanks to the aforementioned properties, Leashed-SGD provides significant improvements in stability as well as wall-clock time to convergence (from 20-80% up to 4× improvements) compared to the standard lock-based AsyncSGD algorithm and HOGWILD!, while reducing the overall memory footprint.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: The Leashed-sgd Frameworkmentioning

confidence: 99%

Consistent Lock-free Parallel Stochastic Gradient Descent for Fast and Stable Convergence

Bäckström¹,

Walulya²,

Papatriantafilou³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Note that any future RangeScan with sequence number i that traverses v while att is still in progress, will see that v is flagged and find out the required information to complete op in its Info object. In PNB-BST, 32 new := pointer to a new Leaf node whose key field is k, its seq field is equal to seq, and its prev field is ⊥ 162 newSibling := pointer to a new Leaf whose key is l → key, its prev field is equal to ⊥ and its seq field is equal to seq 163 newInternal := pointer to a new Internal node with key field max(k, l → key), update field Flag, Dummy , its seq field equal to seq and its prev field equal to l, and with two child fields equal to new and newSibling (the one with the smaller key is the left child), 164 if the RangeScan helps op complete before it continues its traversal.…”

Section: Overviewmentioning

confidence: 99%

“…There has been much recent work on designing efficient concurrent implementations of set data structures [4,5,8,10,12,13,21,29,36,38], which provide algorithms for Insert, Delete, and Find. There is increasing interest in providing additional operations for modern applications, including iterators [1,32,33,35,36,37] or general range queries [6,9]. These are required in many big-data applications [11,26,34], where shared in-memory tree-based data indices must be created for fast data retrieval and useful data analytics.…”

Section: Introductionmentioning

confidence: 99%

Persistent Non-Blocking Binary Search Trees Supporting Wait-Free Range Queries

Fatourou

Papavasileiou

Ruppert

2019

The 31st ACM Symposium on Parallelism in Algorithms and Architectures

View full text Add to dashboard Cite

This paper presents the first implementation of a search tree data structure in an asynchronous shared-memory system that provides a wait-free algorithm for executing range queries on the tree, in addition to non-blocking algorithms for Insert, Delete and Find, using single-word Compareand-Swap (CAS). The implementation is linearizable and tolerates any number of crash failures.Insert and Delete operations that operate on different parts of the tree run fully in parallel (without any interference with one another). We employ a lightweight helping mechanism, where each Insert, Delete and Find operation helps only update operations that affect the local neighbourhood of the leaf it arrives at. Similarly, a RangeScan helps only those updates taking place on nodes of the part of the tree it traverses, and therefore RangeScans operating on different parts of the tree do not interfere with one another. Our implementation works in a dynamic system where the number of processes may change over time.The implementation builds upon the non-blocking binary search tree implementation presented by Ellen et al. [13] by applying a simple mechanism to make the tree persistent. 15 type Internal { subtype of Node 16 Key ∪ {∞1, ∞2} key 17 Update update 18 Node *left, *right 19 Node *prev 20 int seq 21 } 22 type Leaf { subtype of Node 23 Key ∪ {∞1, ∞2} key 24 Update update 25 Node *prev 26 int seq 27 } 28Initialization: 29 shared counter Counter := 0 30 shared Info *Dummy := pointer to a new Info object whose state field is Abort, and whose other fields are ⊥ 31 shared Internal *Root := pointer to new Internal node with key field ∞2, update field Flag, Dummy , prev field ⊥, seq field 0, and its lef t and right fields pointing to new Leaf nodes whose prev fields are ⊥, seq fields are 0, and keys ∞1 and ∞2, respectively 33 Precondition: seq ≥ 0 Precondition: p is non-⊥ and p → seq ≤ seq 45 if lef t then l := p → left else l := p → right Move down to appropriate child 46 while (l → seq > seq) l := l → prev 47 return l; 48 } 49 ValidateLink(Internal *parent, Internal *child, Boolean lef t): Boolean, Update { 50 Preconditions: parent and child are non-⊥ 51 Update up 52 up := parent → update 53 if Frozen(up) then { 54 Help(up.inf o) 55 return False, ⊥ 56 } 57 if (lef t and child = parent → lef t) or (¬lef t and child = parent → right) then return False, ⊥ 58 else return True, up 59 } Boolean validated 64 validated, pupdate := ValidateLink(p, l, k < p → key) 65 if validated and p = Root then validated, gpupdate := ValidateLink(gp, p, k < gp → key) 66 validated := validated and p → update = pupdate and (p = Root or gp → update = gpupdate) 67 return validated, gpupdate, pupdate 68 } 69 Find(Key k): Leaf* { 70 Internal * gp, p 71 Leaf *l 72 Boolean validated 73 while True { 74 seq := Counter 75 −, p, l := Search(k, seq) 76 validated, −, − := ValidateLeaf(gp, p, l, k) 77 if validated then { 78 if l → key = k then return l 79 else return ⊥ 80 } 81} 82 } 83 CAS-Child(Internal *parent, Node *old, Node *new) { Precondition: parent points to an Int...

show abstract

“…Petrank and Timnat [13] proposed a wait-free iterator for non-blocking linked lists [4,14] and lock-free skip lists [6]. Nikolakopoulos et al [12] considered different consistency specifications of iterators for concurrent data structures. In particular, they presented both obstruction-free [5] and wait-free implementations of iterators for the classic lock-free concurrent queue by Michael and Scott [10].…”

Section: Related Workmentioning

confidence: 99%

Linearizable Iterators for Concurrent Sets

Agarwal¹,

Liu²,

Rosenthal³

et al. 2017

Preprint

View full text Add to dashboard Cite

This paper proposes a general framework for adding linearizable iterators to a class of data structures that implement set operations. We introduce a condition on set operations, called local consistency, which informally states that set operations never make elements unreachable to a sequential iterator's traversal. We show that sets with locally consistent operations can be augmented with a linearizable iterator via the framework. Our technique is broadly applicable to a variety of data structures, including hash tables and binary search trees. We apply the technique to sets taken from existing literature, prove their operations are locally consistent, and demonstrate that iterators do not significantly affect the performance of concurrent set operations.

show abstract

A Consistency Framework for Iteration Operations in Concurrent Data Structures

Cited by 12 publications

References 28 publications

Consistent Lock-free Parallel Stochastic Gradient Descent for Fast and Stable Convergence

Consistent Lock-free Parallel Stochastic Gradient Descent for Fast and Stable Convergence

Persistent Non-Blocking Binary Search Trees Supporting Wait-Free Range Queries

Linearizable Iterators for Concurrent Sets

Contact Info

Product

Resources

About