In this paper, we systematically study property testing of unitary operators. We first introduce a distance measure that reflects the average difference between unitary operators. Then we show that, with respect to this distance measure, the orthogonal group, quantum juntas (i.e. unitary operators that only nontrivially act on a few qubits of the system) and Clifford group can be all efficiently tested. In fact, their testing algorithms have query complexities independent of the system's size and have only one-sided error. Then we give an algorithm that tests any finite subset of the unitary group, and demonstrate an application of this algorithm to the permutation group. This algorithm also has one-sided error and polynomial query complexity, but it is unknown whether it can be efficiently implemented in general.