The Noisy Expectation–maximization Algorithm

Osoba, Osonde A.; Mitaim, Sanya; Kosko, Bart

doi:10.1142/s0219477513500120

Cited by 30 publications

(34 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Findings for noiseenhanced hidden Markov models and speech recognition were reported in [124]. The positive effects of noisy was also shown to speed up the average convergence of the ExpectationMaximization (EM) algorithm [125], and centroid-based clustering algorithms such as K-means algorithm [126], under certain conditions. One interesting application of randomization of inputs is in the complexity analysis of algorithms.…”

Section: Noise-enhanced Search Algorithmsmentioning

confidence: 93%

Noise-Enhanced Information Systems

Chen

Varshney

2014

Proc. IEEE

View full text Add to dashboard Cite

Abstract-Noise, traditionally defined as an unwanted signal or disturbance, has been shown to play an important constructive role in many information processing systems and algorithms. This noise enhancement has been observed and employed in many physical, biological, and engineered systems. Indeed stochastic facilitation (SF) has been found critical for certain biological information functions like detection of weak, subthreshold stimuli or suprathreshold signals through both experimental verification and analytical model simulations.In this paper, we present a systematic noise-enhanced information processing framework to analyze and optimize the performance of engineered systems. System performance is evaluated not only in terms of signal-to-noise ratio but also in terms of other more relevant metrics such as probability of error for signal detection or mean square error for parameter estimation.As an important new instance of SF, we also discuss the constructive effect of noise in associative memory recall. Potential enhancement of image processing systems via the addition of noise is discussed with important applications in biomedical image enhancement, image denoising and classification.

show abstract

Section: Noise-enhanced Search Algorithmsmentioning

confidence: 93%

Noise-Enhanced Information Systems

Chen

Varshney

2014

Proc. IEEE

View full text Add to dashboard Cite

show abstract

“…The EM algorithm generalizes the backpropagation algorithm of modern neural networks and the k ‐means clustering algorithm along with many other iterative algorithms. Carefully injected noise always speeds EM convergence on average with the largest gains in the early steps up the hill of likelihood.…”

Section: Mixture‐based Properties Of Additive Fuzzy Systemsmentioning

confidence: 99%

Additive Fuzzy Systems: From Generalized Mixtures to Rule Continua

Kosko

2018

Int. J. Intell. Syst.

Self Cite

View full text Add to dashboard Cite

A generalized probability mixture density governs an additive fuzzy system. The fuzzy system's if‐then rules correspond to the mixed probability densities. An additive fuzzy system computes an output by adding its fired rules and then averaging the result. The mixture's convex structure yields Bayes theorems that give the probability of which rules fired or which combined fuzzy systems fired for a given input and output. The convex structure also results in new moment theorems and learning laws and new ways to both approximate functions and exactly represent them. The additive fuzzy system itself is just the first conditional moment of the generalized mixture density. The output is a convex combination of the centroids of the fired then‐part sets. The mixture's second moment defines the fuzzy system's conditional variance. It describes the inherent uncertainty in the fuzzy system's output due to rule interpolation. The mixture structure gives a natural way to combine fuzzy systems because mixing mixtures yields a new mixture. A separation theorem shows how fuzzy approximators combine with exact Watkins‐based two‐rule function representations in a higher‐level convex sum of the combined systems. Two mixed Gaussian densities with appropriate Watkins coefficients define a generalized mixture density such that the fuzzy system's output equals any given real‐valued function if the function is bounded and not constant. Statistical hill‐climbing algorithms can learn the generalized mixture from sample data. The mixture structure also extends finite rule bases to continuum‐many rules. Finite fuzzy systems suffer from exponential rule explosion because each input fires all their graph‐cover rules. The continuum system fires only a special random sample of rules based on Monte Carlo sampling from the system's mixture. Users can program the system by changing its wave‐like meta‐rules based on the location and shape of the mixed densities in the mixture. Such meta‐rules can help mitigate rule explosion. The meta‐rules grow only linearly with the number of mixed densities even though the underlying fuzzy if‐then rules can have high‐dimensional if‐part and then‐part fuzzy sets.

show abstract

“…The Noisy EM (NEM) algorithm [14,[21][22][23]] is a noise-enhanced version of the EM algorithm that carefully selects noise and then adds it to the data. NEM converges faster on average than EM does because on average it takes larger steps up the same hill of probability or of log-likelihood.…”

Section: Noise Boosting the Expectation-maximization Algorithmmentioning

confidence: 99%

“…The NEM positivity condition (4) holds when the noise-perturbed likelihood f (y + N, z|θ k ) is larger on average than the noiseless likelihood f (y, z|θ k ) at the k th step of the algorithm [21,23]. This noise-benefit condition has a simple quadratic form when the data or signal model is a mixture of Gaussian pdfs.…”

Section: Noise Boosting the Expectation-maximization Algorithmmentioning

confidence: 99%

The Noisy Expectation-Maximization Algorithm for Multiplicative Noise Injection

Osoba

Kosko

2016

Fluct. Noise Lett.

Self Cite

View full text Add to dashboard Cite

We generalize the noisy expectation-maximization (NEM) algorithm to allow arbitrary modes of noise injection besides just adding noise to the data. The noise must still satisfy a NEM positivity condition. This generalization includes the important special case of multiplicative noise injection. A generalized NEM theorem shows that all measurable modes of injecting noise will speed the average convergence of the EM algorithm if the noise satisfies a generalized NEM positivity condition. This noise-benefit condition has a simple quadratic form for Gaussian and Cauchy mixture models in the case of multiplicative noise injection. Simulations show a multiplicative-noise EM speed-up of more than 27% in a Gaussian mixture model. Injecting blind noise only slowed convergence. A related theorem gives a sufficient condition for an average EM noise benefit for arbitrary modes of noise injection if the data model comes from the general exponential family of probability density functions. A final theorem shows that injected noise slows EM convergence on average if the NEM inequalities reverse and the noise satisfies a negativity condition. Noise Boosting the Expectation-Maximization AlgorithmWe show how carefully chosen and injected multiplicative noise can speed convergence of the popular expectation-maximization (EM) algorithm. Multiplicative noise [1] occurs in many applications in signal processing and communications. These include synthetic aperture radar [2-5], sonar imaging [6,7], photonics [8], and random amplitude modulation [9]. A more general theorem allows arbitrary modes of combining signal and noise. The result still speeds EM convergence on average at each iteration so long as the injected noise satisfies a positivity condition.

show abstract

The Noisy Expectation–maximization Algorithm

Cited by 30 publications

References 63 publications

Noise-Enhanced Information Systems

Noise-Enhanced Information Systems

Additive Fuzzy Systems: From Generalized Mixtures to Rule Continua

The Noisy Expectation-Maximization Algorithm for Multiplicative Noise Injection

Contact Info

Product

Resources

About