With the widespread uptake of 2D and 3D single molecule localization microscopy, a large set of different data analysis packages have been developed to generate super-resolution images. In a large community effort we designed a competition to extensively characterise and rank the performance of 2D and 3D single molecule localization microscopy software packages. We generated realistic simulated datasets for popular imaging modalities-2D, astigmatic 3D, biplane 3D, and double helix 3D-and evaluated 36 participant packages against these data. This provides the first broad assessment of 3D single molecule localization microscopy software and provides a holistic view of how the latest 2D and 3D single molecule localization software perform in realistic conditions. This resource allows researchers to identify optimal analytical software for their experiments, allows 3D SMLM software developers to benchmark new software against current state of the art, and provides insight into the current limits of the field. RESULTS Competition design We established a broad committee from the SMLM community, including experimentalists and software developers, to define the scope of the challenge, ensure realism of the datasets and define analysis metrics. We opened this discussion to all interested parties in an online discussion forum 17. In 2016, we ran a first round of the 3D SMLM competition with explicit submission deadlines, culminating in a special session at the 6th annual Single Molecule Localization Microscopy Symposium (SMLMS 2016). Since then, the challenge has been opened to continuously accept new entries. Thirtysix software packages have been entered in the competition thus far, including four packages used in commercial software (Table S1, Supplementary Note 1). Participation in the competition actually led at least eight teams to modify their software to support additional 3D SMLM modalities, showing how competition can foster microscopy software development. Realistic 3D simulations Testing super-resolution software on experimental data lacks the ground truth information required for rigorous quantification of software performance. Therefore, realistic simulated datasets are required. A critical challenge to in simulating 3D SMLM data was accurate modeling of the

In the version of this article originally published online, Saurabh Saha and Gerrit Meijer's names were misspelled. The errors have been corrected in the print, PDF, and HTML versions of the article.