Signal retrieval from a series of indirect measurements is a common task in many imaging, metrology, and characterization platforms in science and engineering. Because most of the indirect measurement processes are well-described by physical models, signal retrieval can be solved with an iterative optimization that enforces measurement consistency and prior knowledge on the signal. These iterative algorithms are time-consuming and only accommodate a linear measurement process and convex signal constraints. Recently, neural networks have been widely adopted to supersede iterative methods by directly approximating the inverse mapping of the measurement process. However, such vanilla network with a deterministic multi-layer structure is unable to distinguish signal ambiguities in ill-posed measurement systems, and the retrieved signals often lack consistency with the measurement. In this work, we incorporate the known measurement process into a customized variational generative model to capture the distribution of all possible signals given a measurement, which can be either a linear or nonlinear process. Our signal retrieval framework resolves the ambiguity in the measurement process, and retrieves high-fidelity signals that satisfy the physical model in a variety of nonlinear, ill-posed systems, such as image retrieval from Fresnel hologram and ultrafast pulse retrieval. INDEX TERMS Variational generative model, computational imaging, neural networks, inverse problem.