Robust visual data segmentation: Sampling from distribution of model parameters

Sadri, Alireza; Tennakoon, Ruwan; Hosseinnezhad, Reza; Bab–Hadiashar, Alireza

doi:10.1016/j.cviu.2018.07.001

Cited by 4 publications

(3 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If this is set to True, certainRatio will be reduced gradually until the number of optimization iterations reaches a predefined value (optIters). This enables the function to deal with skewed probability density functions (Sadri et al, 2018).…”

Section: Functions For Robust Geometric Model Fittingmentioning

confidence: 99%

A Python package based on robust statistical analysis for serial crystallography data processing

Hadian-Jazi

Sadri

2023

Acta Cryst Sect D Struct Biol

View full text Add to dashboard Cite

The term robustness in statistics refers to methods that are generally insensitive to deviations from model assumptions. In other words, robust methods are able to preserve their accuracy even when the data do not perfectly fit the statistical models. Robust statistical analyses are particularly effective when analysing mixtures of probability distributions. Therefore, these methods enable the discretization of X-ray serial crystallography data into two probability distributions: a group comprising true data points (for example the background intensities) and another group comprising outliers (for example Bragg peaks or bad pixels on an X-ray detector). These characteristics of robust statistical analysis are beneficial for the ever-increasing volume of serial crystallography (SX) data sets produced at synchrotron and X-ray free-electron laser (XFEL) sources. The key advantage of the use of robust statistics for some applications in SX data analysis is that it requires minimal parameter tuning because of its insensitivity to the input parameters. In this paper, a software package called Robust Gaussian Fitting library (RGFlib) is introduced that is based on the concept of robust statistics. Two methods are presented based on the concept of robust statistics and RGFlib for two SX data-analysis tasks: (i) a robust peak-finding algorithm and (ii) an automated robust method to detect bad pixels on X-ray pixel detectors.

show abstract

Section: Functions For Robust Geometric Model Fittingmentioning

confidence: 99%

A Python package based on robust statistical analysis for serial crystallography data processing

Hadian-Jazi

Sadri

2023

Acta Cryst Sect D Struct Biol

View full text Add to dashboard Cite

show abstract

“…Such a density function would be a convolution of many kinds of noise functions, including a Poisson density (due to counting photons) convolved with the noise density of the detector that changes with temperature. It was shown by Sadri et al (2018) that the optimization method is capable of finding the mode of a skewed density function (suitable for dealing with Poisson processes). After the mode is detected, an i.i.d (independent and identically distributed) Gaussian noise is assumed for all data points in X to find inliers, and the non-robust average and standard deviation of the inliers (with noise levels less than a given threshold) is given as the output of the robust estimator function.…”

Section: Appendix a Robust Gaussian Fittingmentioning

confidence: 99%

Automatic bad-pixel mask maker for X-ray pixel detectors with application to serial crystallography

Sadri¹,

Hadian-Jazi²,

Yefanov³

et al. 2022

J Appl Cryst

Self Cite

View full text Add to dashboard Cite

X-ray crystallography has witnessed a massive development over the past decade, driven by large increases in the intensity and brightness of X-ray sources and enabled by employing high-frame-rate X-ray detectors. The analysis of large data sets is done via automatic algorithms that are vulnerable to imperfections in the detector and noise inherent with the detection process. By improving the model of the behaviour of the detector, data can be analysed more reliably and data storage costs can be significantly reduced. One major requirement is a software mask that identifies defective pixels in diffraction frames. This paper introduces a methodology and program based upon concepts of machine learning, called robust mask maker (RMM), for the generation of bad-pixel masks for large-area X-ray pixel detectors based on modern robust statistics. It is proposed to discriminate normally behaving pixels from abnormal pixels by analysing routine measurements made with and without X-ray illumination. Analysis software typically uses a Bragg peak finder to detect Bragg peaks and an indexing method to detect crystal lattices among those peaks. Without proper masking of the bad pixels, peak finding methods often confuse the abnormal values of bad pixels in a pattern with true Bragg peaks and flag such patterns as useful regardless, leading to storage of enormous uninformative data sets. Also, it is computationally very expensive for indexing methods to search for crystal lattices among false peaks and the solution may be biased. This paper shows how RMM vastly improves peak finders and prevents them from labelling bad pixels as Bragg peaks, by demonstrating its effectiveness on several serial crystallography data sets.

show abstract

“…The success of the above optimization algorithm depends on the input parameter k (Sadri et al, 2018). It should be below the possible number of outliers in any window around Bragg peaks.…”

Section: Robust Model Fittingmentioning

confidence: 99%

Data reduction for serial crystallography using a robust peak finder

Hadian-Jazi¹,

Sadri²,

Barty³

et al. 2021

J Appl Crystallogr

Self Cite

View full text Add to dashboard Cite

A peak-finding algorithm for serial crystallography (SX) data analysis based on the principle of `robust statistics' has been developed. Methods which are statistically robust are generally more insensitive to any departures from model assumptions and are particularly effective when analysing mixtures of probability distributions. For example, these methods enable the discretization of data into a group comprising inliers (i.e. the background noise) and another group comprising outliers (i.e. Bragg peaks). Our robust statistics algorithm has two key advantages, which are demonstrated through testing using multiple SX data sets. First, it is relatively insensitive to the exact value of the input parameters and hence requires minimal optimization. This is critical for the algorithm to be able to run unsupervised, allowing for automated selection or `vetoing' of SX diffraction data. Secondly, the processing of individual diffraction patterns can be easily parallelized. This means that it can analyse data from multiple detector modules simultaneously, making it ideally suited to real-time data processing. These characteristics mean that the robust peak finder (RPF) algorithm will be particularly beneficial for the new class of MHz X-ray free-electron laser sources, which generate large amounts of data in a short period of time.

show abstract

Robust visual data segmentation: Sampling from distribution of model parameters

Cited by 4 publications

References 54 publications

A Python package based on robust statistical analysis for serial crystallography data processing

A Python package based on robust statistical analysis for serial crystallography data processing

Automatic bad-pixel mask maker for X-ray pixel detectors with application to serial crystallography

Data reduction for serial crystallography using a robust peak finder

Contact Info

Product

Resources

About