Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms

Abstract

Although acknowledged to be variable and subjective, manual annotation of cryo-electron tomography data is commonly used to answer structural questions and to create a “ground truth” for evaluation of automated segmentation algorithms. Validation of such annotation is lacking, but is critical for understanding the reproducibility of manual annotations. Here, we used voxel-based similarity scores for a variety of specimens, ranging in complexity and segmented by several annotators, to quantify the variation among their annotations. In addition, we have identified procedures for merging annotations to reduce variability, thereby increasing the reliability of manual annotation. Based on our analyses, we find that it is necessary to combine multiple manual annotations to increase the confidence level for answering structural questions. We also make recommendations to guide algorithm development for automated annotation of features of interest.

Cryo-soft X-ray tomography: using soft X-rays to explore the ultrastructure of whole cells

“…This process can be entirely manual, with a person outlining the boundaries of cellular structures of interest on each slice through the 3D volume [81,84]; or segmentation algorithms can be used to decrease the time and manual input required [19,[85][86][87]. Manual segmentation can be, by far, the most time-consuming step in a cryo-SXT experiment and it is generally regarded as somewhat objective and variable [88]. Recent advances in data representation and segmentation have been introduced at Diamond Light Source through the SuRVoS Workbench, which uses minimal manual training inputs to assign voxels in the volume to user-defined classes [19,89].…”

The advent of structural biology in situ by single particle cryo-electron tomography

“…Reliable automation of tiltseries reconstruction into tomograms will also increase the throughput of studies by cryoET (Mastronarde and Held 2017). Automated segmentation of features in tomograms remains poor and can be extremely subjective and time consuming when performed manually (Hecksel et al 2016), particularly for thick frozen-hydrated specimens. The use of compressed sensing and other methodologies that improve the quality of tomographic reconstructions should facilitate more accurate particle picking and more objective, automated annotation of cellular tomograms.…”

Volume Segmentation and Analysis of Biological Materials Using SuRVoS (Super-region Volume Segmentation) Workbench

“…As imaging technologies and segmentation strategies improve, it has become more common to use segmentation as a quantitative tool to answer biological questions and as a "ground truth" for algorithm development 8 12 15 23 24 25 . In order to do this, detailed checks and balances are required to decrease variability and subjectivity throughout the process 26 . However, these precautions further increase the time-consuming nature of segmentation.…”