Labeling Poststorm Coastal Imagery for Machine Learning: Measurement of Interrater Agreement

Abstract

Key points:1) We measure agreement among coastal scientists labeling the same sets of post-storm images.2) Coastal scientists agree more when rating landforms, less when labeling inferred processes.3) Iterating on questions, providing documentation, and using smaller image sizes all increase agreement.

Human‐in‐the‐Loop Segmentation of Earth Surface Imagery

“…The two examples shown in Figure 8e with relatively poor agreement do so for different reasons; in the upper example the two labelers have disagreed over the two shadow classes, and in the lower example the two labelers have disagreed where one identifies a region as coarse whereas the other identifies it as wood. In these examples, consensus could be achieved through some rules‐based process, or by redoing the labels with lower‐than‐average IOU and/or Dice scores in order to achieve greater label precision through consensus (Goldstein et al., 2021; Monarch, 2021).…”

“…For example, in the sidescan data set (data set D), the distribution of per‐class scores has the largest range; shadow and wood classes achieve relatively little consensus (Figure 13b). The two shadow classes would likely have to be merged for consistency, and better agreement over wood and all the other categories might be possible if a manual documenting examples is prepared (Goldstein et al., 2021). In the post‐hurricane data set (data set B), sand is often difficult to distinguish from water for the same reasons as described for data set.…”

“…Supervised ML will therefore continue to be popular, and powerful, if facilitated by open‐source tools that make data labeling more efficient, and analyses of uncertainty that add vital context to its use. Doodler, as what Monarch (2021) refers to as a “smart interface for semantic segmentation,” is one of many specific software tools or interfaces (Bueno et al., 2020; Goldstein et al., 2021; Zhao et al., 2020) for the generation of large labeled data sets (Kashinath, Mudigonda, et al., 2021; Sumbul et al., 2019) that can be used for teaching and self‐exploration of Deep Learning techniques, for use in transfer learning, and for new model development. Doodler is an open‐source program that runs in a web browser, and may be one of many similar future implementations that might use human‐in‐the‐loop ML for efficient labeling of other scientifically relevant label data such as those generated from time‐series signals or social media content (Cai et al., 2017).…”

“…This section serves a few purposes. First, for subjective tasks involving interpretation of ambiguous data, or even objective tasks or relatively simple tasks where random human blunder may be a factor, no simple heuristics exist for deciding the correct label (Monarch, 2021) however some practical recommendations can be made using statistical metrics of multi‐labeled datasets (Goldstein et al., 2021). Similarly, we offer some methods for identifying and quantifying uncertainty based on agreement over segmentations of the same imagery by multiple labelers.…”

The future of coastal monitoring through satellite remote sensing

“…As satellite datasets become bigger the application of modern machine-learning modeling workflows to evaluate Earth surface processes becomes increasingly attractive, particularly as data science workflows become more robust, scalable, and accessible through open-source software (e.g., Morgan et al, 2019;Gibeaut et al, 2019;Goldstein et al, 2021a;Demir et al, 2022;Buscombe et al, in press, Sun et al, 2022). Machine learning is broadly defined as teaching a computer algorithm to learn by example.…”