Statistical Performance Comparisons of Computers

Guo, Qi; Temam, Olivier; Wu, Yue; Bao, Yungang; Xu, Zhiwei; Chen, Tianshi

doi:10.1109/tc.2014.2315614

Cited by 31 publications

(29 citation statements)

References 33 publications

(19 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mogwaï is a model query framework that tackles this issue for models stored in NeoEMF/Graph, translating OCL queries to Tinkerpop Gremlin through ATL and reporting reductions in execution up to a factor of 20 [11]. 9 http://wiki.eclipse.org/Teneo/Hibernate. 10 https://github.com/BryanHunt/mongo-emf/wiki.…”

Section: Database-backed Model Persistencementioning

confidence: 99%

“…At the same time, 10% of those benchmarks were not normally distributed (being reportedly skewed or bimodal), and modern machines with multi-core processors have only grown more non-deterministic since then. More recently, Chen et al [9] concluded that execution times for multithreaded loads in modern multi-core machines do not follow neither normal nor log-normal distributions and that more robust nonparametric methods are needed for performance comparison. Our study in particular involves 3 machines communicating over Ethernet and doing heavy disk-based I/O: even with 1 thread per client machine, the server will experience a nondeterministic multithreaded load as the one studied by Chen at al.…”

Section: Measurements Obtainedmentioning

confidence: 99%

See 1 more Smart Citation

Stress-testing remote model querying APIs for relational and graph-based stores

et al. 2017

View full text Add to dashboard Cite

Recent research in scalable model-driven engineering now allows very large models to be stored and queried. Due to their size, rather than transferring such models over the network in their entirety, it is typically more efficient to access them remotely using networked services (e.g. model repositories, model indexes). Little attention has been paid so far to the nature of these services, and whether they remain responsive with an increasing number of concurrent clients. This paper extends a previous empirical study on the impact of certain key decisions on the scalability of concurrent model queries on two domains, using an Eclipse Connected Data Objects model repository, four configurations of the Hawk model index and a Neo4j-based configuration of the NeoEMF model store. The study evaluates the impact of the network protocol, the API design, the caching layer, the query language and the type of database and analyses the reasons for their varying levels of performance. The design of the API was shown to make a bigger difference compared to the network protocol (HTTP/TCP) used. Where available, the query-specific indexed and derived attributes in Hawk outperformed the comprehensive generic caching in CDO. Finally, the results illustrate the still ongoing evolution of graph databases: two tools using different versions of the same backend had very different performance, with one slower than CDO and the other faster than it.

show abstract

Section: Database-backed Model Persistencementioning

confidence: 99%

Section: Measurements Obtainedmentioning

confidence: 99%

Stress-testing remote model querying APIs for relational and graph-based stores

et al. 2017

View full text Add to dashboard Cite

show abstract

“…To motivate the discussion, we analyze the overhead caused by emulation in a dynamic binary translator (DynamoRIO [4]) when running SPECint 2006 using the Hierarchical Performance Test (HPT) [7]. The quality of a DBT (Dynamic Bynary Translator) is measured by its fitness to translate computer programs efficiently and transparently [3,10].…”

Section: Motivationmentioning

confidence: 99%

“…Furthermore, when two data files are provided, the tool is able to compare the measurements of the given files using different metrics such as the HPTspeedup and the geometric mean of ratios (e.g. SPECrate) [7,11]. The tool implements the HPT framework, which is not available in conventional tools such as R [16].…”

Section: Motivationmentioning

confidence: 99%

“…More recently, Chen et al [7] argued that parametric statistical techniques should not be used when assessing computer systems performance data and introduced a hierarchical performance test framework (HPT) that applies consecutive statistical hypothesis tests on the performance scores of two computing systems. The framework reports how much a system is faster than the other and with which confidence level.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Assessing computer performance with stocs

Piga

Gomes

Auler

et al. 2013

Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

Several aspects of a computer system cause performance measurements to include random errors. Moreover, these systems are typically composed of a non-trivial combination of individual components that may cause one system to perform better or worse than another depending on the workload. Hence, properly measuring and comparing computer systems performance are non-trivial tasks.The majority of work published on recent major computer architecture conferences do not report the random errors measured on their experiments. The few remaining authors have been using only confidence intervals or standard deviations to quantify and factor out random errors. Recent publications claim that this approach could still lead to misleading conclusions.In this work, we reproduce and discuss the results obtained in a previous study. Finally, we propose SToCS, a tool that integrates several statistical frameworks and facilitates the analysis of computer science experiments.

show abstract

The Impact of Dynamic Storage Allocation on CPython Execution Time, Memory Footprint and Energy Consumption: An Empirical Study

Lamprakos

Papadopoulos

Catthoor

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

CPython is the reference implementation of the Python programming language, as well as the most popular one. Tools like machine learning frameworks, web development interfaces and scientific computing libraries have been built on top of it. Meanwhile, singleboard computers are now able to run GNU/Linux distributions. As a result, CPython's influence today is not limited to commodity servers, but also includes edge and mobile devices. We should thus be concerned with the performance of CPython applications. In this spirit, we investigate the impact of dynamic storage allocation on the execution time, memory footprint and energy consumption of CPython programs. Our findings show that (i) CPython's default configuration is optimized for memory footprint, (ii) replacing this configuration can improve performance by up to 1.135x and (iii) application-specific characteristics define which allocator setup performs best at each case. Additionally, we contribute an open-source means of benchmarking the energy consumption of CPython applications, which we implemented for our experiments. Last but not least, by employing a rigorous and reliable statistical analysis technique, we provide strong indicators that our conclusions are platform-independent.

show abstract

Statistical Performance Comparisons of Computers

Abstract: Abstract

Cited by 31 publications

References 33 publications

Stress-testing remote model querying APIs for relational and graph-based stores

Stress-testing remote model querying APIs for relational and graph-based stores

Assessing computer performance with stocs

The Impact of Dynamic Storage Allocation on CPython Execution Time, Memory Footprint and Energy Consumption: An Empirical Study

Contact Info

Product

Resources

About