Scattering is defined on compact manifolds with boundary which are equipped with an asymptotically hyperbolic metric, g. A model form is established for such metrics close to the boundary. It is shown that the scattering matrix at energy ζ exists and is a pseudo-differential operator of order 2ζ + 1 − dim X. The symbol of the scattering matrix is then used to show that except for a countable set of energies the scattering matrix at one energy determines the diffeomorphism class of the metric modulo terms vanishing to infinite order at the boundary. An analogous result is proved for potential scattering. The total symbol is computed when the manifold is hyperbolic or is of product type modulo terms vanishing to infinite order at the boundary. The same methods are then applied to studying inverse scattering on the Schwarzschild and De Sitter-Schwarzschild models of black holes.