This thesis presents methods for vision-based obstacle detection and traversability estimation in field environments. The first challenge addressed by this research is that vegetation obscures the ground and obstacles, which limits structure-based methods. The second challenge addressed is that the large variance in environments, illumination conditions and obstacles limits appearance-based methods.Vegetation is a challenge for traversability estimation, as it obscures the ground. Traversability estimation requires complete and accurate knowledge of the ground height. This thesis presents a method to infer in real-time the ground height in the presence of vegetation, which is shown to significantly reduce the ground height estimation error in vegetation. The method is used with velocity modulation to reduce peak vertical acceleration during real-time operation.Vegetation is also a challenge for structure-based obstacle detection methods, as it obscures the obstacles. The large variance in environments, illumination conditions and obstacles are challenges for appearance-based obstacle detection methods, as it makes it infeasible to pre-train an obstacle detector. Some obstacles may be camouflaged in their appearance or their structure, and so a combination of the two is required to detect all obstacles. This thesis presents methods for real-time obstacle detection which utilise both appearance and structure cues to detect novel obstacles.These methods are capable of adapting in real-time to environmental and illumination changes. The results demonstrate that these methods are able to detect a wide variety of static and dynamic obstacles during field trials in a variety of environments at day and night.Vision-based methods can be improved via the design of image descriptors which are highly invariant to illumination changes. To improve the performance of invariant image descriptors it is necessary to be able to quantify the illumination variance of image descriptors. This thesis presents novel metrics for quantifying the illumination variance of descriptors. The results demonstrate that the metrics are suitable for quantifying the illumination variance of image descriptors, and make recommendations about the most invariant image descriptors.