Traditionally, cryptanalysis of optical security systems attempts to find original keys. Usually, by use of this kind of method, one can find a set of keys located close to the original keys in the key space. We call such a set the region of original key (ROK). For an optical encryption system in the Fresnel domain, such a strategy is ineffective since it needs to perform an exhaustive search to determine the system geometry or to solve an extremely large set of system equations. We propose to employ an alternative search strategy: to find a region of possible key (RPK). Since there is only one ROK for a cypher system but there are many RPKs, the probability to find a key in the RPK would be higher than in the ROK. Our analysis reveals that even a Fresnel-based encryption system has larger key space, but there are also serious security problems to be resolved.