The time evolution of a two-component collisionless plasma is modeled by the Vlasov-Poisson system. In this work, the setting is two and one-half dimensional, that is, the distribution functions of the particles species are independent of the third space dimension. We consider the case that an external magnetic field is present in order to confine the plasma in a given infinitely long cylinder. After discussing global well-posedness of the corresponding Cauchy problem, we construct stationary solutions which indeed have support away from their confinement device. Then, in the main part of this work we investigate the stability of such steady states, both with respect to perturbations in the initial data, where we employ the energy-Casimir method, and also with respect to perturbations in the external magnetic field.