Efficient collective communication in distributed heterogeneous systems

Abstract

The Information Power Grid (IPG)

Toward understanding heterogeneity in computing

“…In 1994, the first author noted the need for better understanding of the scheduling implications of heterogeneity via rigorous analyses [23]. There has since been an impressive amount of first-rate work on this topic-focusing largely on collective communication [3,4,8,15,17,22,24], but also studying important scheduling issues [1,5,6,7,13,18]. That said, sources such as [1] show that there is still much to learn about this important topic-including the questions in the abstract.…”

On Multicast Algorithms for Heterogeneous Networks of Workstations

“…Thus the total running time is O(n log n). L Although the greedy algorithm is intuitively appealing, it is known to produce nonoptimal multicast schedules [1,3]. Figure 1a shows the schedule found by the greedy algorithm, completing at time 10.…”

“…In related work, Bhat et al have proposed an alternative model that accounts for heterogeneity in both the nodes and the network [3]. This model is particularly well-suited for wide-area networks where network latencies over ''long haul'' links may be very different from those within a local area network.…”

Pipelining broadcasts on heterogeneous platforms

“…Since the message size is a constant in the case of a broadcast, the total communication time between P i and P j is C i;j ¼ T i;j þ m B i;j . In [22], some heuristics are proposed for the broadcast and the multicast using this model.…”

“…Yet, another model of communication is introduced in [22], [23]: the time needed to transfer the message between any processor pair ðP i ; P j Þ is supposed to be divided into a start-up cost T i;j and a part depending on the size m of the message and the transmission rate B i;j between the two processors, m B i;j . Since the message size is a constant in the case of a broadcast, the total communication time between P i and P j is C i;j ¼ T i;j þ m B i;j .…”