We examine in this work the role of disorder in contributing to the sluggish relaxation observed in intrinsic electron-glasses. Our approach is guided by several empirical observations: First and foremost, Anderson localization is a pre-requisite for observing these nonequilibrium phenomena. Secondly, sluggish relaxation appears to favor Anderson-insulators with relatively large Fermi-energies (hence proportionally large disorder). These observations motivated us to consider a way to measure the underlying disorder in a realistic Anderson insulator. Optical study using a series of amorphous indium-oxide (InxO) establish a simple connection between carrier-concentration and the disorder necessary to approach the metal-insulator transition from the insulating side. This is used to estimate the typical magnitude of the quenched potential-fluctuation in the electron-glass phase of this system. The implications of our findings on the slow dynamics of Anderson-insulators are discussed. In particular, the reason for the absence of a memory-dip and the accompanying electron-glass effects in lightly-doped semiconductors emerges as a natural consequence of their weak disorder.