Vacuum polarization (or Casimir) energies can be straightforwardly computed from scattering data for static field configurations whose interactions with the fluctuating field are frequency independent. In effective theories, however, such interactions are typically frequency dependent. As a consequence, the relationship between scattering data and the Green's function is modified, which may or may not induce additional contributions to the vacuum polarization energy. We discuss several examples that naturally include frequency dependent interactions: (i) scalar electrodynamics with a static background potential, (ii) an effective theory that emerges from integrating out a heavy degree of freedom, and (iii) quantum electrodynamics coupled to a frequency dependent dielectric material. In the latter case, we argue that introducing dissipation as required by the Kramers-Kronig relations requires the consideration of the Casimir energy within a statistical mechanics formalism, while in the absence of dissipation we can work entirely within field theory, using an alternative formulation of the energy density.