We analyze the expectation value of the Polyakov loop in the fundamental and higher representations in the confined phase of QCD. We discuss a hadronic like representation, and find that the Polyakov loop corresponds to a partition function in the presence of a colored source, explaining its real and positive character. Saturating the sum rules to intermediate temperatures requires a large number of multipartonic excited states. By using constituent or bag models, we find detailed low temperature scaling rules which depart from the Casimir scaling and could be tested by lattice calculations.