Scipion3: A workflow engine for cryo-electron microscopy image processing and structural biology

Abstract

Image-processing pipelines require the design of complex workflows combining many different steps that bring the raw acquired data to a final result with biological meaning. In the image-processing domain of cryo-electron microscopy single-particle analysis (cryo-EM SPA), hundreds of steps must be performed to obtain the three-dimensional structure of a biological macromolecule by integrating data spread over thousands of micrographs containing millions of copies of allegedly the same macromolecule. The execution of such complicated workflows demands a specific tool to keep track of all these steps performed. Additionally, due to the extremely low signal-to-noise ratio (SNR), the estimation of any image parameter is heavily affected by noise resulting in a significant fraction of incorrect estimates. Although low SNR and processing millions of images by hundreds of sequential steps requiring substantial computational resources are specific to cryo-EM, these characteristics may be shared by other biological imaging domains. Here, we present Scipion, a Python generic open-source workflow engine specifically adapted for image processing. Its main characteristics are: (a) interoperability, (b) smart object model, (c) gluing operations, (d) comparison operations, (e) wide set of domain-specific operations, (f) execution in streaming, (g) smooth integration in high-performance computing environments, (h) execution with and without graphical capabilities, (i) flexible visualization, (j) user authentication and private access to private data, (k) scripting capabilities, (l) high performance, (m) traceability, (n) reproducibility, (o) self-reporting, (p) reusability, (q) extensibility, (r) software updates, and (s) non-restrictive software licensing.

Scipion-EM-ProDy: A Graphical Interface for the ProDy Python Package within the Scipion Workflow Engine Enabling Integration of Databases, Simulations and Cryo-Electron Microscopy Image Processing

“…The general idea behind Scipion is to integrate many software packages into complex workflows comprised of various interconnected protocols, whose outputs are the inputs to subsequent protocols [ 33 ]. These protocols are provided by plugins, including the Scipion core plugins [ 33 ], as well many others for external software, such as Xmipp v3.23.07 [ 66 ], Relion 3.1 and 4.0 [ 67 ], ContinuousFlex 3.4.0 [ 13 ] and ChimeraX 1.0 to 1.4 [ 19 ]. A key feature is interoperability by handling data in common objects and converting it between different formats required for different programs.…”

“…In this context, we recently created the Scipion Flexibility Hub [ 32 ], a framework for combining continuous flexibility methods into pipelines with shared interactive analysis tools connected to the rest of the Scipion workflow engine for Cryo-EM image analysis and structural biology (version 3.0) [ 33 , 34 ]. This framework provides common tools for several continuous heterogeneity methods, including the CryoDRGN deep reconstruction generative network (supporting version 1.0.0, 1.1.0, 2.1.0 and 2.3.0) [ 11 ], HEMNMA [ 24 , 35 ], MDSPACE [ 27 ], and related methods presented in ContinuousFlex v3.4.0 [ 13 ], and some approaches based on the Zernike3D deformation framework [ 12 , 36 ].…”

MDSPACE and MDTOMO Software for Extracting Continuous Conformational Landscapes from Datasets of Single Particle Images and Subtomograms Based on Molecular Dynamics Simulations: Latest Developments in ContinuousFlex Software Package

“…MDTOMO is an MDSPACE extension to continuous conformational landscape extraction from 3D subtomogram data, which was fully described in [ 35 ]. The NMMD, MDSPACE, and MDTOMO methods were implemented in ContinuousFlex [ 47 ], which can be run as a plugin of Scipion [ 48 , 49 ].…”

“…If prior atomic structural information is available, these methods allow obtaining the conformational landscape at atomic scale (an atomic model is obtained for each particle image or subtomogram, beside 3D density map reconstructions from different regions of the landscape). These methods are available as part of the open-source software package ContinuousFlex [ 47 ] (starting from version 3.4.2 of ContinuousFlex), which is also available as a plugin for Scipion [ 48 , 49 ] and part of ScipionTomo and Scipion Flexibility Hub frameworks [ 50 , 51 ].…”

Scipion-Chem: An Open Platform for Virtual Drug Screening

“…Scipion-chem is an extension of the Scipion workflow engine − that has been designed to get rid of these complications by (1) automatically installing the software of interest and building any necessary environment to avoid conflicts; (2) automatically converting the intermediate files using OBabel, RDKit, or Biopython when necessary; (3) offering a graphic user interface (GUI) to design and manage the workflows and running the programs; (4) saving and organizing all the parameters, workflows, and files in a straightforward folder structure; and (5) providing intuitive viewers and consensus protocols to compare and extract the most relevant information out of results from different software in equivalent steps.…”