Purpose: In this work we design and validate a model observer that can detect groups of microcalcifications in a four alternative forced choice (4-AFC) experiment and use it to optimize a smoothing prior for detectability of microcalcifications.Methods: A channelized Hotelling observer (CHO) with eight Laguerre-Gauss channels was designed to detect groups of five microcalcifications in a background of acrylic spheres by adding the CHO log-likelihood ratios calculated at the expected locations of the five calcifications.This model observer is then applied to optimize the detectability of the microcalcifications as a function of the smoothing prior. We examine the quadratic and total variation (TV) priors, and a combination of both. A selection of these reconstructions was then evaluated by human observers to validate the correct working of the model observer.Results: We found a clear maximum for the detectability of microcalcification when using the total variation prior with weight βT V = 35. Detectability only varied over a small range for the quadratic and combined quadratic-TV priors when weight βQ of the quadratic prior was changed by two orders of magnitude.Spearman correlation with human observers was good except for the highest value of β for the quadratic and TV priors. Excluding those, we found ρ = 0.93 when comparing detection fractions, and ρ = 0.86 for the fitted detection threshold diameter.Conclusions: We successfully designed a model observer that was able to predict human performance over a large range of settings of the smoothing prior, except for the highest values of β which were outside the useful range for good image quality.Since detectability only depends weakly on the strength of the combined prior, it is not possible to pick an optimal smoothness based only on this criterion. On the other hand, such choice can now be made based on other criteria without worrying about calcification detectability.