A Benchmark Data Set and Evaluation of Deep Learning Architectures for Ball Detection in the RoboCup SPL

Abstract

This paper presents a benchmark data set for evaluating ball detection algorithms in the RoboCup Soccer Standard Platform League. We created a labelled data set of images with and without ball derived from vision log files recorded by multiple NAO robots in various lighting conditions. The data set contains 5209 labelled ball image regions and 10924 non-ball regions. Non-ball image regions all contain features that had been classified as a potential ball candidate by an existing ball detector. The data set was used to train and evaluate 252 different Deep Convolutional Neural Network (CNN) architectures for ball detection. In order to control computational requirements, this evaluation focused on networks with 2-5 layers that could feasibly run in the vision and cognition cycle of a NAO robot using two cameras at full frame rate (2×30 Hz). The results show that the classification performance of the networks is quite insensitive to the details of the network design including input image size, number of layers and number of outputs at each layer. In an effort to reduce the computational requirements of CNNs we evaluated XNOR-Net architectures which quantize the weights and activations of a neural network to binary values. We examined XNOR-Nets corresponding to the real-valued CNNs we had already tested in order to quantify the effect on classification metrics. The results indicate that ball classification performance degrades by 12% on average when changing from real-valued CNN to corresponding XNOR-Net.

A Lightweight Region Proposal Network for Task Specific Applications

“…We will then show that this property is also reflected in our real-world dataset for autonomous robot soccer [6].…”

“…We performed the same calculation on our own SPL dataset for autonomous robot soccer [6], which consists of 4,140 images containing 12,048 object instances (3,973 ball, 3,944 robot, 2,534 goal post, and 1,597 penalty spot). The dataset was gathered from a range of Nao robot camera image logs in various locations and lighting conditions including our lab, RoboCup2016 indoor and outdoor, and RoboCup 2017.…”

Real-Time Scene Understanding Using Deep Neural Networks for RoboCup SPL

“…The vision pipelines used by the competing teams have changed in tandem, going from human-engineered vision methods [31,32] to pipelines relying increasingly on machine learning. Several teams have used convolutional neural networks either for binary classification tasks [7,8] or to detect several relevant object categories [9,10]. These methods, however, use CNNs for classification only, therefore they still require a separate object proposal method, and the quality of the system may largely depend on the efficiency of the algorithm used to generate candidates for classification.…”

“…Due to the rapidly increasing power of hardware, the usually computationally expensive networks have started to appear in mobile and embedded systems [6]. Several teams [7][8][9][10][11] have used convolutional neural networks in the 2017 SPL league in RoboCup to classify relevant objects on the soccer field. However, due to the limitations of the robot's hardware, these networks were relatively shallow and were designed to classify fixed-resolution image patches only.…”

Evaluating Extended Pruning on Object Detection Neural Networks

“…The SPL network is based on tiny-YOLO and trained on a custom dataset taken from the robot soccer problem domain [21]. The dataset consists of four distinct classes, ball, robot, goal post, and penalty spot.…”