Guidelines for reporting results of computational experiments. Report of the ad hoc committee

Boggs, Paul T.; Nash, Stephen G.; Powell, Susan; Jackson, Richard H F

doi:10.1007/bf01588801

Cited by 61 publications

(33 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It was actually providing some suggestions as to how the community could be more rigorous in reporting its experiments and findings. Jackson et al (1991) say that "Controversy often surrounds the reporting of results from scientific experimentation. Subtle points with unrecognized but profound effects are sometimes overlooked, and testing procedures are sometimes inadequate."…”

Section: Optimization Researchmentioning

confidence: 99%

Good Laboratory Practice for optimization research

Kendall

Bai

Błażewicz

et al. 2016

Journal of the Operational Research Society

View full text Add to dashboard Cite

show abstract

Section: Optimization Researchmentioning

confidence: 99%

Good Laboratory Practice for optimization research

Kendall

Bai

Błażewicz

et al. 2016

Journal of the Operational Research Society

View full text Add to dashboard Cite

show abstract

“…On the other hand, there are a number of reports about comparing codes in local optimization [1,4,6,7,13,16,27], and there is an extensive web site [21] with wide-ranging comparative results on local constrained optimization codes. Section 3 describes the tests done on the most important state of the art global solvers.…”

Section: Overviewmentioning

confidence: 99%

A comparison of complete global optimization solvers

et al. 2005

View full text Add to dashboard Cite

Abstract. Results are reported of testing a number of existing state of the art solvers for global constrained optimization and constraint satisfaction on a set of over 1000 test problems in up to 1000 variables. OverviewAs the recent survey Neumaier [24] of complete solution techniques in global optimization documents, there are now about a dozen solvers for constrained global optimization that claim to solve global optimization and/or constraint satisfaction problems to global optimality by performing a complete search.Within the COCONUT project [30,31], we evaluated many of the existing software packages for global optimization and constraint satisfaction problems. This is the first time that different constrained global optimization and constraint satisfaction algorithms are compared on a systematic basis and with a test set that allows to derive statistically significant conclusions. We tested the global solvers BARON, GlobSol, ICOS, LGO, LINGO, OQNLP, Premium Solver, the local solver MINOS, and a basic combination strategy COCOS implemented in the COCONUT platform. 1The testing process turned out to be extremely time-consuming, due to various reasons not initially anticipated. A lot of effort went into creating appropriate interfaces, making the comparison fair and reliable, and making it possible to process a large number of test examples in a semiautomatic fashion.In a recent paper about testing local optimization software, Dolan & Moré [6,7] write: We realize that testing optimization software is a notoriously difficult problem and that there may be objections to the testing presented in this report. For example, performance of a particular solver may improve significantly if non-default options are given. Another objection is that we only use one starting point per problem and that the performance of a solver may be sensitive to the choice of starting point. We also have used the default stopping criteria of the solvers. This choice may bias results but should not affect comparisons that rely on large time differences. In spite of these objections, we feel that it is essential that we provide some indication of the performance of optimization solvers on interesting problems.These difficulties are also present with our benchmarking studies. Section 2 describes our testing methodology. We use a large test set of over 1000 problems from various collections. Our main performance criterium is currently how often the attainment of the global optimum, or the infeasibility of a problem, is correctly or incorrectly claimed (within some time limit). All solvers are tested with the default options suggested by the providers of the codes, with the request to stop at a time limit or after the solver believed that first global solution was obtained.These are very high standards, much more demanding than what had been done by anyone before. Thorough comparisons are indeed very rare, due to the difficulty of performing extensive and meaningful testing. Indeed, we know of only two comparative studies [17,22] in global ...

show abstract

“…The authors paid attention to reproducibility of the results and provided a method for reporting the results. In 1990, similar research conducted by Jackson et al [JBNP90] delivered an updated set of guidelines. In 2002, Dolan and Moré introduced performance profiles [DM02], which have rapidly become a gold standard in benchmarking of optimization algorithms with more recent work pointing out its limitations [GS16].…”

Section: Historical Overview Of Benchmarking In Optimizationmentioning

confidence: 98%

Best practices for comparing optimization algorithms

2017

View full text Add to dashboard Cite

The final publication is available at Springer via http://dx.doi.org/10.1007/s11081-017-9366-1 Comparing, or benchmarking, of optimization algorithms is a complicated task that involves many subtle considerations to yield a fair and unbiased evaluation. In this paper, we systematically review the benchmarking process of optimization algorithms, and discuss the challenges of fair comparison. We provide suggestions for each step of the comparison process and highlight the pitfalls to avoid when evaluating the performance of optimization algorithms. We also discuss various methods of reporting the benchmarking results. Finally, some suggestions for future research are presented to improve the current benchmarking process.

show abstract

Guidelines for reporting results of computational experiments. Report of the ad hoc committee

Cited by 61 publications

References 18 publications

Good Laboratory Practice for optimization research

Good Laboratory Practice for optimization research

A comparison of complete global optimization solvers

Best practices for comparing optimization algorithms

Contact Info

Product

Resources

About