This paper describes the use of polarisation information for surface segmentation based on material characteristics. We work with both polarised and unpolarised light, and hence domains where the polarisation is either specular or diffuse. We commence by using moments to estimate the components of the polarisation image (mean-intensity, polarisation degree and phase) from images obtained through multiple polariser orientations. From the Fresnel theory, the phase of light remitted from a surface is equal to the azimuth angle of the remitted direction, and for materials with restricted ranges of refractive index the polarisation degree determines the zenith angle. Based on this observation, we parameterise the angular distribution of the mean intensity for remitted light using spherical harmonics. We explore how vectors of spherical harmonics can be used to characterise varying surface reflectance distributions, and segment a scene into different material patches using Mahalanobis distances and normalized graph cuts.