The polarization analysis of the light is typically carried out using modulation schemes. The light of unknown polarization state is passed through a set of known modulation optics and a detector is used to measure the total intensity passing the system. The modulation optics is modified several times and, with the aid of such several measurements, the unknown polarization state of the light can be inferred. How to find the optimal demodulation process has been investigated in the past. However, since the modulation matrix has to be measured for a given instrument and the optical elements can present problems of repeatability, some uncertainty is present in the elements of the modulation matrix and/or covariances between these elements. We analyze in detail this issue, presenting analytical formulae for calculating the covariance matrix produced by the propagation of such uncertainties on the demodulation matrix, on the inferred Stokes parameters and on the efficiency of the modulation process. We demonstrate that, even if the covariance matrix of the modulation matrix is diagonal, the covariance matrix of the demodulation matrix is, in general, non-diagonal because matrix inversion is a nonlinear operation. This propagates through the demodulation process and induces correlations on the inferred Stokes parameters.