Structure occurs over a vast range of scales in the universe. Our large-scale cosmological models are coarse-grained representations of what exists, which have much less structure than there really is. An important problem for cosmology is determining the influence the small-scale structure in the universe has on its large-scale dynamics and observations. Is there a significant, general relativistic, backreaction effect from averaging over structure? One issue is whether the process of smoothing over structure can contribute to an acceleration term and so alter the apparent value of the cosmological constant. If this is not the case, are there other aspects of concordance cosmology that are affected by backreaction effects? Despite much progress, this 'averaging problem' is still unanswered, but it cannot be ignored in an era of precision cosmology, for instance it may affect aspects of Baryon Acoustic Oscillation observations.

Light from 'point sources' such as supernovae is observed with a beam width of the order of the sources' size -typically less than 1 au. Such a beam probes matter and curvature distributions that are very different from coarse-grained representations in N-body simulations or perturbation theory, which are smoothed on scales much larger than 1 au. The beam typically travels through unclustered dark matter and hydrogen with a mean density much less than the cosmic mean, and through dark matter haloes and hydrogen clouds. Using N-body simulations, as well as a Press-Schechter approach, we quantify the density probability distribution as a function of beam width and show that, even for Gpc-length beams of 500 kpc diameter, most lines of sight are significantly underdense. From this we argue that modelling the probability distribution for au-diameter beams is absolutely critical. Standard analyses predict a huge variance for such tiny beam sizes, and non-linear corrections appear to be non-trivial. It is not even clear whether underdense regions lead to dimming or brightening of sources, owing to the uncertainty in modelling the expansion rate which we show is the dominant contribution. By considering different reasonable approximations which yield very different cosmologies, we argue that modelling ultra-narrow beams accurately remains a critical problem for precision cosmology. This could appear as a discordance between angular diameter and luminosity distances when comparing supernova observations to baryon acoustic oscillations or cosmic microwave background distances.

We present a derivation of the cosmological distance-redshift relation up to second order in perturbation theory. In addition, we find the observed redshift and the lensing magnification to second order. We do not require that the density contrast is small, we only that the metric potentials and peculiar velocities are small. Thus our results apply into the nonlinear regime, and can be used for most dark energy models. We present the results in a form which can be readily computed in an N-body simulation. This paper accompanies Paper I, where the key results are summarised in a physically transparent form and applications are discussed.

In this paper, we develop in detail a fully geometrical method for deriving perturbation equations about a spatially homogeneous background. This method relies on the 3 + 1 splitting of the background space-time and does not use any particular set of coordinates: it is implemented in terms of geometrical quantities only, using the tensor algebra package xTensor in the xAct distribution along with the extension for perturbations xPert. Our algorithm allows one to obtain the perturbation equations for all types of homogeneous cosmologies, up to any order and in all possible gauges. As applications, we recover the well-known perturbed Einstein equations for FriedmannLemaître-Robertson-Walker cosmologies up to second order and for Bianchi I cosmologies at first order. This work paves the way to the study of these models at higher order and to that of any other perturbed Bianchi cosmologies, by circumventing the usually too cumbersome derivation of the perturbed equations.