Toward Replicable and Measurable Robotics Research [From the Guest Editors]

Bonsignorio, Fabio; Pobil, Ángel P. del

doi:10.1109/mra.2015.2452073

Cited by 55 publications

(33 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This section proposes standardized cost functions for the motion planning problem in (2). Analogously to the composability of the benchmarks, we compose different types of partial cost functions to a single cost function.…”

Section: Cost Functionsmentioning

confidence: 99%

“…Reproducibility of results is a cornerstone of science [1], [2]. One obstacle towards reproducibility in motion planning of road vehicles is that details of the experimental results are often not fully provided-some reasons are page limitations of publications, an overwhelming number of required details, or simply because some details are taken for granted.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CommonRoad: Composable benchmarks for motion planning on roads

Althoff

Koschi

Manzinger

2017

2017 IEEE Intelligent Vehicles Symposium (IV)

285

166

View full text Add to dashboard Cite

Abstract-Numerical experiments for motion planning of road vehicles require numerous components: vehicle dynamics, a road network, static obstacles, dynamic obstacles and their movement over time, goal regions, a cost function, etc. Providing a description of the numerical experiment precise enough to reproduce it might require several pages of information. Thus, only key aspects are typically described in scientific publications, making it impossible to reproduce results-yet, reproducibility is an important asset of good science. Composable benchmarks for motion planning on roads (CommonRoad) are proposed so that numerical experiments are fully defined by a unique ID; all information required to reconstruct the experiment can be found on the CommonRoad website. Each benchmark is composed by a vehicle model, a cost function, and a scenario (including goals and constraints). The scenarios are partly recorded from real traffic and partly hand-crafted to create dangerous situations. We hope that CommonRoad saves researchers time since one does not have to search for realistic parameters of vehicle dynamics or realistic traffic situations, yet provides the freedom to compose a benchmark that fits one's needs.

show abstract

Section: Cost Functionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

CommonRoad: Composable benchmarks for motion planning on roads

Althoff

Koschi

Manzinger

2017

2017 IEEE Intelligent Vehicles Symposium (IV)

285

166

View full text Add to dashboard Cite

show abstract

“…EURON Good Experimental Methodology (GEM) or the IEEE Technical Committee on Performance Evaluation and Benchmarking of Robotic and Automation Systems (TC-PEBRAS), to develop methodologies and tools that improve the situation. One outcome is the demand by Bonsignorio and del Pobil [1], who call for a possibility to practically replicate results for validation. The authors suggest a new communitywide agreement on the content of a research paper in robotics.…”

Section: Introductionmentioning

confidence: 99%

“…To increase the maturity of robotics research, the assurance of reproducibility and the possibility of objective disconfirmation is indispensable. This paper proposes a novel workflow that addresses the requirements stated by Bonsignorio and del Pobil [1] and introduces a framework that abstracts the complexity of experimental setups by using containerization technologies. The proposed approach makes it significantly easier to share the experimental artifacts and reproduce the environment of the inital experiment.…”

Section: Introductionmentioning

confidence: 99%

SwarmRob: A Docker-Based Toolkit for Reproducibility and Sharing of Experimental Artifacts in Robotics Research

Prtner¹,

Hoffmann²,

Zug³

et al. 2018

2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC)

View full text Add to dashboard Cite

Due to the complexity of robotics, the reproducibility of results and experiments is one of the fundamental problems in robotics research. While the problem has been identified by the community, the approaches that address the problem appropriately are limited. The toolkit proposed in this paper tries to deal with the problem of reproducibility and sharing of experimental artifacts in robotics research by a holistic approach based on operating-system-level virtualization. The experimental artifacts of an experiment are isolated in "containers" that can be distributed to other researchers. Based on this, this paper presents a novel experimental workflow to describe, execute and distribute experimental software-artifacts to heterogeneous robots dynamically. As a result, the proposed solution supports researchers in executing and reproducing experimental evaluations.

show abstract

“…However, in the robotics field reproducing the results or comparing with other published results is difficult in some cases as pointed out in [Bonsignorio and Del Pobil, 2015]. Different hardware, software barebones and sometimes lack of details makes it difficult to reproduce the results of an alternative solution.…”

Section: An Application Perspective Of Benchmarkingmentioning

confidence: 99%

Visibility in underwater robotics: Benchmarking and single image dehazing

Soler¹

View full text Add to dashboard Cite

A mis padres y hermanos. AcknowledgementsThis thesis is the result of more than 5 years of work and it would have been impossible without the support of family, friends and colleagues. May the reader forgive me the discourtesy of following my wish to acknowledge their support in my mother tongue.But before changing to my mother tongue, I want to thank professor Stefan Williams for hosting me at the Australian Centre for Field Robotics in Sydney for four months. Working with his group, and specially with Mitch Bryson and Dushyant Rao, was crucial for the development of the final chapters of this thesis. I also have to acknowledge the ACFR for providing and preparing the datasets used in the deep learning experiments.En primer lugar, me gustaría agradecer a mis directores Raúl Marín y Pedro Sanz la oportunidad de poder trabajar en un campo tan motivante y desafiante como la robótica submarina a través de distintos proyectos en la frontera del conocimiento como Trident, Triton o Merbots. Además de por la ayuda y los consejos desde la experiencia a la hora de introducirme en la investigación.Gracias a la Universitat Jaume I por haber financiado durante 4 años el desarrollo de esta tesis a través del plan de promoción de la investigación (referencia de la ayuda PREDOC/2012/47).También tengo que dar las gracias a todos los miembros del IRSLab de los que he podido aprender, y con los que he colaborado y trabajado como un verdadero equipo durante estos años. A todos quiero dar gracias, Juan Carlos por su disponibilidad a la hora de ayudar, Javi por su espíritu de equipo, Toni por su decisión a la hora de afrontar problemas, David por susánimos y voluntad, Jorge por su insistencia en publicar, José Bernabé por sus consejos y Diego por su entrega.Gracias también a Mario, pese al poco tiempo que coincidimos demostró ser un investigador sobresaliente y un compañero ejemplar que me ayudó en los primeros momentos de este trabajo. Además también tengo que agradecer al RobInLab, especialmente aÁngel, Carlos, Marco, Xavi, Gabriel, Majd y Gustavo, por su ayuda y discusiones en los descansos delante de una taza de café.Durante mi estancia en Sydney tuve la suerte de coincidir con un pequeño grupo de españoles que hizo la estancia mucho más amena. Por ello también quiero darles las gracias, en particular a Guillem y a Raúl con los cuales compartí mucho tiempo dentro y fuera del laboratorio visitando y conociendo Australia, y en general a todas las personas con las que coincidí durante esos meses.Gracias a mi familia y amigos de la Venta del Moro, siempre dispuestos a escuchar después de una larga semana de trabajo. Durante estos años han sido v una distracción necesaria a los problemas del día a día, haciendo el camino de esta tesis mucho más entretenido.Quiero también agradecer a Loli por ser una más de la familia y preocuparse y cuidar de mí día a día desde bien pequeño, ayudándome a convertirme en una persona mejor.Porúltimo, quiero agradecer especialmente la ayuda y el apoyo de mi familia. A mis padres Luis y MaríaÁngeles que m...

show abstract

Toward Replicable and Measurable Robotics Research [From the Guest Editors]

Cited by 55 publications

References 3 publications

CommonRoad: Composable benchmarks for motion planning on roads

CommonRoad: Composable benchmarks for motion planning on roads

SwarmRob: A Docker-Based Toolkit for Reproducibility and Sharing of Experimental Artifacts in Robotics Research

Visibility in underwater robotics: Benchmarking and single image dehazing

Contact Info

Product

Resources

About