Proba-V cloud detection Round Robin: Validation results and recommendations

Cloud Detection: An Assessment Study from the ESA Round Robin Exercise for PROBA-V

“…We also observe that accuracy is strongly dependent on the surface type for all methods; in particular it is higher on Land than on Water. This result is consistent among all methods participating the Round Robin exercise [4] and was there justified with a more accurate training. In this experiment this is equivalent to the fact that Land scenes are 2/3 of Total ones in the global training set, so that classification naturally tends to better represent them.…”

“…We mention that a second dataset of 53,000 scenes was prepared during the ESA project. It served as a basis for comparing algorithms participating in the Round Robin exercise [4]. However, this dataset has never been revealed to participants, even after the end of the exercise [10].…”

“…An extensive analysis through experiments analogous to the ones described in this paper (not reported for the sake of brevity) confirms that accuracy of the combined cloud mask outperforms the single ones. In [4] some comparisons are shown for the gold standard dataset.…”

“…While the main purpose of such exercises is to develop accurate operational algorithms for specific sensors onboard satellites, an important side effect is comparison of state-of-art methods on a same, very accurate dataset. In this respect we mention the Landsat comparison exercise [3] and the ESA Round Robin exercise for PROBA-V sensor [4]. Such comparisons are an exceptional way not only to compare algorithms, but especially to discover their weakness in particular climatic/surface conditions and, finally, to progress knowledge of cloud mask detection.…”

“…The key data of the exercise is a set of 1350 scenes, blind to participants that have been manually labelled by experts, claimed to sample the most important types of clouds (gold standard). A more detailed description of the exercise and main conclusions are reported in [4].…”