A Benchmark Data Set to Evaluate the Illumination Robustness of Image Processing Algorithms for Object Segmentation and Classification

Abstract

Developers of image processing routines rely on benchmark data sets to give qualitative comparisons of new image analysis algorithms and pipelines. Such data sets need to include artifacts in order to occlude and distort the required information to be extracted from an image. Robustness, the quality of an algorithm related to the amount of distortion is often important. However, using available benchmark data sets an evaluation of illumination robustness is difficult or even not possible due to missing ground truth data about object margins and classes and missing information about the distortion. We present a new framework for robustness evaluation. The key aspect is an image benchmark containing 9 object classes and the required ground truth for segmentation and classification. Varying levels of shading and background noise are integrated to distort the data set. To quantify the illumination robustness, we provide measures for image quality, segmentation and classification success and robustness. We set a high value on giving users easy access to the new benchmark, therefore, all routines are provided within a software package, but can as well easily be replaced to emphasize other aspects.

A New Feedback-Based Method for Parameter Adaptation in Image Processing Routines

“…A benchmark data set ( https://sourceforge.net/projects/gait-cad/files/Benchmarks/hardware_items/ ) is specifically designed to conform to evaluation criteria that are most suitable for our methodology. A complete description of data set is given in [ 32 ]. It is based on 4 scenes r = 1, …, 4 containing solid objects i.e.…”

“…Measures based on a given ground truth previously used in [ 32 ] are segmentation measures ( q 1 and q 2 ) and a classification measure ( q 3 ). Segmentation measures penalize the difference between objects detected and the number of non-overlapping pixels with respect to the ground truth.…”

“…The classification measure is based on the number of misclassified pixels with respect to the ground truth. These measures ( q 1 , q 2 and q 3 ) are converted to fuzzy functions using criteria given in [ 32 ] and are represented as μ 1 , μ 2 and μ 3 . An overall quality criterion Q ( r , b , n ) based on measures given in [ 32 ] is: and a robustness measure ( R ) according to [ 32 ] is given as: …”

“…To measure segmentation and classification accuracy, a data set must contain both, segmentation and classification ground truth. Recently, we developed a benchmark data set which contains image scenes with ground truth about object position and object class, varying noise and artifact levels [ 32 ].…”

Generating semi-synthetic validation benchmarks for embryomics

“…It has been shown that biological phenomena such as fluorescently labeled cell populations can be realistically simulated if enough knowledge of the investigated probes was available [5][6][7][8]. The charm of simulated data is the availability of a reliable ground truth and literally unrestricted possibilities to adjust parameters like noise levels, sampling rates or light attenuation, which can hardly be achieved by imaging dynamically changing organisms and thus prohibits robustness analyses as in [9]. Nevertheless, existing simulated benchmarks are often much simpler than the real application scenarios and mostly focus solely on a single processing step.…”