In experiments that involve contact with adhesion between two surfaces, as found in atomic force microscopy or nanoindentation, two distinct contact force (P) vs. indentation-depth (h) curves are often measured depending on whether the indenter moves towards or away from the sample. The origin of this hysteresis is not well understood and is often attributed to moisture, plasticity or viscoelasticity. Here we report experiments that show that hysteresis can exist in the absence of these effects, and that its magnitude depends on surface roughness. We develop a theoretical model in which the hysteresis appears as the result of a series of surface instabilities, in which the contact area grows or recedes by a finite amount. The model can be used to estimate material properties from contact experiments even when the measured P-h curves are not unique.