Topologically ordered systems exhibit large-scale correlation in their ground states, which may be characterized by quantities such as topological entanglement entropy. We propose that the concept of irreducible many-body correlation (IMC), the correlation that cannot be implied by all local correlations, may also be used as a signature of topological order. In a topologically ordered system, we demonstrate that for a part of the system with holes, the reduced density matrix exhibits IMCs which become reducible when the holes are removed. The appearance of these IMCs then represents a key feature of topological phase. We analyze the many-body correlation structures in the ground state of the toric code model in external magnetic fields, and show that the topological phase transition is signaled by the IMCs.