It is important to be able to quantify, theoretically and experimentally, the performance of coherent digital systems, so that their suitability for a given metrology application can be assessed. Here, a free-space inline digital holographic system is investigated. To isolate the scattered object field, phase-shifting interferometry (PSI) techniques are used. Several sequential holographic measurements are made, where the phase of the reference field is stepped by a known amount relative to the scattered object field between captures. Under ideal conditions such as noise-free electronics, vibration-free environments, and perfect reference and illuminating object waves, this system will be diffraction limited. However, real systems suffer from experimental error and noise effects. In this paper, we examine a PSI digital holographic imaging system considering all prominent error sources. An experimental metric is defined that quantifies how far from the theoretical ideal a real system is performing. By carefully optimizing our system, following our recommended guidelines, we approach diffraction limited imaging, surpassing the Nyquist sampling rate of the CCD/CMOS device.