We develop a systematic field theoretic description for the roughness correction to the Casimir free energy of parallel plates. Roughness is modeled by specifying a generating functional for correlation functions of the height profile, the two-point correlation function being characterized by the variance, σ 2 , and correlation length, ℓ, of the profile. We obtain the partition function of a massless scalar quantum field interacting with the height profile of the surface via a δ-function potential. The partition function of this model also is given by a holographic reduction to three coupled scalar fields on a two-dimensional plane. The original three-dimensional space with a flat parallel plate at a distance a from the rough plate is encoded in the non-local propagators of the surface fields on its boundary. Feynman rules for this equivalent 2 + 1-dimensional model are derived and its counter terms constructed. The two-loop contribution to the free energy of this model gives the leading roughness correction. The absolute separation, a eff , to a rough plate is measured to an equivalent plane that is displaced a distance ρ ∝ σ 2 /ℓ from the mean of its profile. This definition of the separation eliminates corrections to the free energy of order 1/a 4 eff and results in a unitary model. We derive an effective low-energy theory in the limit ℓ ≪ a. It gives the scattering matrix and equivalent planar surface of a very rough plate in terms of the single length scale ρ. The Casimir force on a rough plate is found to always weaken with decreasing correlation length ℓ. The two-loop approximation to the free energy interpolates between the free energy of the effective low-energy model and that of the proximity force approximation -the force on a very rough Dirichlet plate with σ 0.5ℓ being weaker than on a flat plate at any separation.