From Data to Uncertainty: An Efficient Integrated Data-Driven Sparse Grid Approach to Propagate Uncertainty

Franzelin, Fabian; Pflüger, Dirk

doi:10.1007/978-3-319-28262-6_2

Cited by 6 publications

(8 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sparse grid clustering, as implemented in the open source library SG ++ , is one the few clustering methods available for the clustering of large datasets on HPC machines. In contrast to other density estimation approaches, our approach based on sparse grids does not depend on assumptions about the underlying densities and which can treat correlated densities [30,31]. In particular, it enables the detection of clusters with non-convex shapes and without a predetermined number of clusters; see [21] for the comparison to other clustering algorithms.…”

Section: Discussion and Future Workmentioning

confidence: 99%

Heterogeneous Distributed Big Data Clustering on Sparse Grids

2019

Self Cite

View full text Add to dashboard Cite

Clustering is an important task in data mining that has become more challenging due to the ever-increasing size of available datasets. To cope with these big data scenarios, a high-performance clustering approach is required. Sparse grid clustering is a density-based clustering method that uses a sparse grid density estimation as its central building block. The underlying density estimation approach enables the detection of clusters with non-convex shapes and without a predetermined number of clusters. In this work, we introduce a new distributed and performance-portable variant of the sparse grid clustering algorithm that is suited for big data settings. Our computed kernels were implemented in OpenCL to enable portability across a wide range of architectures. For distributed environments, we added a manager–worker scheme that was implemented using MPI. In experiments on two supercomputers, Piz Daint and Hazel Hen, with up to 100 million data points in a ten-dimensional dataset, we show the performance and scalability of our approach. The dataset with 100 million data points was clustered in 1198 s using 128 nodes of Piz Daint. This translates to an overall performance of 352 TFLOPS . On the node-level, we provide results for two GPUs, Nvidia’s Tesla P100 and the AMD FirePro W8100, and one processor-based platform that uses Intel Xeon E5-2680v3 processors. In these experiments, we achieved between 43% and 66% of the peak performance across all computed kernels and devices, demonstrating the performance portability of our approach.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Heterogeneous Distributed Big Data Clustering on Sparse Grids

2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, the tensor-product approach is an efficient technique when the number of random variables is small, whereas sparse grid techniques (e.g., the Smolyak algorithm [61]) are proved to be more efficient for a large number of random variables (see Appendix A). Note that various adaptive methods for quadrature node selection have also been introduced in the literature: for example, in [62], an adaptive algorithm based on nested sparse grids is proposed and applied to a stochastic eddy current NDT problems, a weighted Smolyak algorithm is presented in [63], an adaptive hierarchical sparse grid collocation algorithm and a data-driven sparse grid approach are illustrated in [64,65], respectively. In conclusion, a spectral projection can be considered as a nonintrusive strategy, requiring a suitable deterministic problem to be solved K times [23,66].…”

Section: Pc-based Applications In Electronicsmentioning

confidence: 99%

“…More details about tensor products and sparse grids based on the Smolyak algorithm are given in Appendix A. Various adaptive sparse grid methods have been proposed in the literature to further mitigate the curse of dimensionality, such as nested sparse grids [62], weighted Smolyak algorithms [63], hierarchical [64] and data-driven [65] sparse grids, and the dimension adaptive approach [94].…”

Section: Efficient Sampling Strategies For High-dimensional Problemsmentioning

confidence: 99%

Review of Polynomial Chaos-Based Methods for Uncertainty Quantification in Modern Integrated Circuits

2018

View full text Add to dashboard Cite

Advances in manufacturing process technology are key ensembles for the production of integrated circuits in the sub-micrometer region. It is of paramount importance to assess the effects of tolerances in the manufacturing process on the performance of modern integrated circuits. The polynomial chaos expansion has emerged as a suitable alternative to standard Monte Carlo-based methods that are accurate, but computationally cumbersome. This paper provides an overview of the most recent developments and challenges in the application of polynomial chaos-based techniques for uncertainty quantification in integrated circuits, with particular focus on high-dimensional problems.

show abstract

“…In the context of UQ, additional options arise to address the outer loop optimisation, because many black-box computer model runs are required. There exist already various techniques such as surrogate models [12][13][14], multifidelity models [15], model order reduction [16], sparse grid interpolation or cubature [17][18][19]. But all these techniques at the end require numerous black-box computer model runs where idling can occur.…”

Section: Introductionmentioning

confidence: 99%

Prediction and reduction of runtime in non-intrusive forward UQ simulations

2019

View full text Add to dashboard Cite

To foster predictive simulations, a variety of methods have recently been developed to efficiently tackle uncertainty quantification (UQ) in complex, computational intensive problems. Many of these methods are non-intrusive and, thus, result in a (large) number of embarrassingly parallel black-box evaluations of the underlying simulation codes. While the focus of development is typically on the number of black-box evaluations, which represents the bulk of the computational workload, an additional level of potential performance gains exists. In many scenarios, uncertain input leads not only to uncertain outputs, but also to a varying and thus stochastic runtime of the simulation codes. For scheduling the individual black-box runs, this information is typically not taken into account, resulting in non-negligible idling times on parallel systems. In this contribution, we compare a variety of different scheduling strategies for non-intrusive UQ scenarios using the non-intrusive polynomial chaos approach. In particular, we propose to construct a surrogate model for the runtime of the application using the identical UQ methodology as for the original problem. Using this model to predict the runtimes for subsequent black-box runs allows for (heuristical) optimization of the scheduling. The method has been tested for the forward quantification of uncertainty on academic models and on a pedestrian simulation in the context of evacuation scenarios. This approach allows speed-up factors of about two for the total runtime and can be generalised to a large variety of applications that incorporate parameter-dependent runtime.

show abstract

From Data to Uncertainty: An Efficient Integrated Data-Driven Sparse Grid Approach to Propagate Uncertainty

Cited by 6 publications

References 30 publications

Heterogeneous Distributed Big Data Clustering on Sparse Grids

Heterogeneous Distributed Big Data Clustering on Sparse Grids

Review of Polynomial Chaos-Based Methods for Uncertainty Quantification in Modern Integrated Circuits

Prediction and reduction of runtime in non-intrusive forward UQ simulations

Contact Info

Product

Resources

About