We study the vacuum (i.e., zero-temperature) Casimir energy for a system of neutral conductors which are isolated , as opposed to grounded. The former is meant to describe a situation where the total charge on each conductor, as well as all of its fluctuations, vanishes, while the latter describes a situation where the conductors are connected to a charge reservoir. We compute the difference between the vacuum energies for a given system of conductors, but subjected to the two different conditions stated above. The results can be written in terms of a generalized, frequency-dependent capacitance matrix of the system. Using a multipolar expansion, we show that the grounded Casimir energy includes a monopole-monopole interaction term that is absent in the isolated case in the large distance limit.