Hyper-parameter optimization tools comparison for multiple object tracking applications

Abstract

Commonly, when developing an algorithm it is necessary to define a certain number of variables that control its behavior. Optimal parameters result in better performance that could translate into profits for companies, stand out among similar applications or better ranking on algorithm competitions. However, it is not a simple task to find the combination of parameters that provides the best results. Manual tuning could be a stressful and difficult task even for expert users. In this paper we present, evaluate and compare several tools in the literature for hyper-parameter optimization. We focus on 4 tools that have been selected due to their number of citations, code availability and impact on literature: MCMC, SMAC, TPE and Spearmint. We analyze these tools in the context of Multi Object Tracking (MOT) that have not been well studied in the literature. MOT itself has been well-studied topic with multiple parameters to be tuned. We evaluate these tools using public benchmarks such as PETS09 or ETH and using the publicly available source code provided by the authors. We analyze the impact of these tools in terms of stability, performance, and usability, among others. Our results show how the use of these tools change the performance of the application and how this would affect the results of real ranked competitions. Our goal is (1) to encourage the reader to use these tools and (2) to provide a guide that helps to choose the most pertinent tool.

Integrated Simulation-Based Optimization of Operational Decisions at Container Terminals

“…Bergstra et al [41] quickly followed with an alternative approach that they called TPE. A comparison between several data sets showed that the performance of such a hyper-parameter optimization technique varies with each setup [42][43][44]. This research topic is often referred to as hyper-parameter optimization and is the subject of active research and development [44][45][46].…”

“…In the past several years, many newly developed hyper-parameter optimization approaches have advanced the field. Studies such as [42][43][44] have empirically compared different meta-heuristics. This approach is the key to identifying characteristics of each meta-heuristic, which, in turn, can result in the recommendation of one meta-heuristic.…”

“…However, when several optimization studies are considered together, the observed characteristics of each optimization algorithm (e.g., a meta-heuristic) should fit into the broader picture. Hence, a comparison study such as [42][43][44] or this publication contributes to these deeper insights.…”

Integrated Simulation-based Optimization of Operational Decisions at Container Terminals

“…Bergstra et al [36] quickly followed with an alternative approach and called it TPE. A comparison between several data-sets showed that the performance of such a hyper-parameter optimization technique varies with each setup [37][38][39]. The research topic is often referred to as hyper-parameter optimization and is subject to active research and development [39][40][41].…”

“…Comparison studies such as [37][38][39] are very crucial to identify strengths and weaknesses of these approaches. Sörensen [42] admonishes that the design and application of new meta-heuristics should go beyond playing a simple up-the-wall game.…”