In a gauge theory, the gauge invariant Hilbert space is unchanged by the coupling to arbitrary local operators. In the presence of Wilson loops, though, the physical Hilbert space must be enlarged by adding test electric charges along the loop. I discuss how at nonzero temperature Polyakov loops are naturally related to the propagator of a test charge. 't Hooft loops represent the propagation of a test magnetic charge, and so do not alter the physical Hilbert space.