In complex systems, crucial parameters are often subject to unpredictable changes in time. Climate, biological evolution and networks provide numerous examples for such non-stationarities. In many cases, improved statistical models are urgently called for. In a general setting, we study systems of correlated quantities to which we refer as amplitudes. We are interested in the case of non-stationarity, i.e., seemingly random covariances. We present a general method to derive the distribution of the covariances from the distribution of the amplitudes. To ensure analytical tractability, we construct a properly deformed Wishart ensemble of random matrices. We apply our method to financial returns where the wealth of data allows us to carry out statistically significant tests. The ensemble that we find is characterized by an algebraic distribution which improves the understanding of large events.