A benchmark dataset for individual tree crown delineation in co-registered airborne RGB, LiDAR and hyperspectral imagery from the National Ecological Observation Network

Abstract

Broad scale remote sensing promises to build forest inventories at unprecedented scales. A crucial step in this process is designing individual tree segmentation algorithms to associate pixels into delineated tree crowns. While dozens of tree delineation algorithms have been proposed, their performance is typically not compared based on standard data or evaluation metrics, making it difficult to understand which algorithms perform best under what circumstances. There is a need for an open evaluation benchmark to minimize differences in reported results due to data quality, forest type and evaluation metrics, and to support evaluation of algorithms across a broad range of forest types. Combining RGB, LiDAR and hyperspectral sensor data from the National Ecological Observatory Network’s Airborne Observation Platform with multiple types of evaluation data, we created a novel benchmark dataset to assess individual tree delineation methods. This benchmark dataset includes an R package to standardize evaluation metrics and simplify comparisons between methods. The benchmark dataset contains over 6,000 image-annotated crowns, 424 field-annotated crowns, and 3,777 overstory stem points from a wide range of forest types. In addition, we include over 10,000 training crowns for optional use. We discuss the different evaluation sources and assess the accuracy of the image-annotated crowns by comparing annotations among multiple annotators as well as to overlapping field-annotated crowns. We provide an example submission and score for an open-source baseline for future methods.

BAMFORESTS: Bamberg Benchmark Forest Dataset of Individual Tree Crowns in Very-High-Resolution UAV Images

“…Nevertheless, BAMFORESTS is still a relatively small dataset for application in deep learning, but projects like Million Trees from Weinstein [16] will hopefully solve this in the future. Furthermore, the labels in BAMFORESTS are not of perfect quality.…”

“…The Million Trees project [16] aims to collect as many existing datasets covering all above-mentioned categories and creating a combined benchmark dataset of at least one million labeled trees. As BAMFORESTS falls into the category of a very-high-resolution datasets with complete polygon labels of single trees, we summarize similar datasets in Table 1.…”

“…Therefore, only a few small, openly accessible datasets exist. The Million Trees project [16] initiated by Weinstein et al [17] tries to solve this by collecting and combining as many available datasets as possible. Nevertheless, this is an ongoing effort which, according to Ben Weinstein, will take place until fall 2025.…”