Process Affinity, Metrics and Impact on Performance: An Empirical Study

Abstract

Process placement, also called topology mapping, is a well-known strategy to improve parallel program execution by reducing the communication cost between processes. It requires two inputs: the topology of the target machine and a measure of the affinity between processes. In the literature, the dominant affinity measure is the communication matrix that describes the amount of communication between processes. The goal of this paper is to study the accuracy of the communication matrix as a measure of affinity. We have done an extensive set of tests with two fat-tree machines and a 3d-torus machine to evaluate several hypotheses that are often made in the literature and to discuss their validity. First, we check the correlation between algorithmic metrics and the performance of the application. Then, we check whether a good generic process placement algorithm never degrades performance. And finally, we see whether the structure of the communication matrix can be used to predict gain.Key-words: process placement, topology mapping, MPI, communication, algorithm, communication modeling, performance metric Affinité entre les processus, métriques et impact sur les performances : étude expérimentale Résumé : Le placement de processus en prenant en compte la topologie de la machine est une technique bien connue pour réduire le temps d'exécution d'un programme parallèle en diminuant le coût des communications entre les processus. Il nécessite deux entrées : la topologie de la machine cible, et une mesure de l'affinité entre les processus. Dans la littérature, la mesure d'affinité qui prédomine est la matrice de communication qui comptabilise les communications entre les processus. Le but de ce papier est d'étudier la pertinence de la matrice de communication comme mesure de l'affinité. Dans ce but, nous avons réalisé un grand nombre de tests sur une machine de type fat-tree ainsi que sur un tore 3d, afin d'évaluer plusieurs hypothèse qui se retrouvent souvent dans la littérature et de discuter de leur validité. Pour cela, d'abord nous vérifions la corrélation entre des métriques algorithmiques et la performance de l'application. Ensuite, nous contrôlons qu'un bon algorithme de placement n'implique jamais une dégradation des performances d'une application. Et finalement, nous étudions la structure de la matrice de communication dans le but de voir si elle peut être utilisée dans la prédiction du gain.

Mapping Matters: Application Process Mapping on 3-D Processor Topologies

“…Process-logical communication matrices provide a first view into an application's communication behavior. The structure of a communication matrix alone may prove insufficient to predict if and how much will an application benefit from careful process mapping [18]. To enable such predictions, communication metrics (or matrix-based statistics) have previously been proposed and tested in the context of thread mapping on shared-memory machines [22,23] and process mapping on hierarchical machines (multicore nodes in fat-tree topologies) [18].…”

“…The structure of a communication matrix alone may prove insufficient to predict if and how much will an application benefit from careful process mapping [18]. To enable such predictions, communication metrics (or matrix-based statistics) have previously been proposed and tested in the context of thread mapping on shared-memory machines [22,23] and process mapping on hierarchical machines (multicore nodes in fat-tree topologies) [18]. Such metrics are useful when we are interested in predicting performance gains through mapping, but also to triage applications for which mapping studies can not be performed (e.g., when multiple applications need to be tested while access to resources is very limited) or for which mapping is not expected to yield performance improvements.…”

“…The following communication metrics are considered in this work: Communication Amount (CA) indicates the average inter-process communication [23]; Communication Balance (CB ) measures the divergence of the most communicating process from the others [22]; Communication Centrality (CC ) quantifies the dispersion of communication from the main diagonal in the communication matrix [18]; Communication Heterogeneity (CH ) denotes the average communication variance of each process [23]; Neighbor Communication Fraction (NBC ) represents the fraction of communication between processes with close rank identifiers (nearest neighbors) [18]; and Split Fraction SP(k) measures the fraction of communication among processes in k 2 blocks [18].…”

“…For all metrics, higher values indicate a higher potential benefit via careful process mapping. We computed the values of the six above metrics for the four applications using the formulae [7] proposed in [18] except for CA, which is not mentioned therein and, thus, we obtained it independently in this work. We computed the Split Fraction of applications for k = 4 and k = 16 as they represent a portion and a full plane in the 3-D topologies considered, respectively (more details in Section 5).…”

“…In this work, we use dilation as a pre-simulation metric to evaluate the mappings' quality. Dilation is widely used as a performance and quality metric in process mapping [11,18,31,54], where it is also referred to as 'hop-Bytes'. Dilation is easy to determine and tends to correlate well with the applications' performance [15].…”