Mechanical oscillators are extensively used in applications ranging from chronometry using quartz oscillators to ultra fast sensors and actuators as in atomic force microscopy and spin/mass/force sensing. The continuous push towards miniaturized high precision sensors has motivated the development of increasingly high quality mechanical oscillators at length scales as small as nanometers, with integration into cryogenic environments used to avoid thermal noise. This raises the prospect of observing mechanical oscillators in a new regime where their dynamics are dictated by quantum, rather than classical, mechanics. Utilizing the quantum behaviour of mechanical oscillators promises to enhance applications in sensing and metrology. In this thesis experimental techniques are reported that enhance optomechanical sensors and enable fundamental experiments in quantum optomechanics. In particular, there is a strong emphasis towards experimentally developing detection based feedback control techniques, for example to enable feedback cooling towards the mechanical ground state or to stabilize parasitic instabilities. In addition to these experimental advances, I detail a real-time estimation strategy that invalidates the use of linear feedback to enhanced the performance of linear optomechanical sensors. Finally, a unique and promising optomechanical systems is developed and characterized based on surface waves of a thin film of superfluid helium-4.The first topic considered in this thesis is feedback cooling of a generalized optomechanical system. In that chapter a detailed mathematical treatment is derived that highlights the importance of using a dual probe position measurement to accurately characterize a feedback cooled oscillator. The theoretical results are then experimentally verified using a microtoroidal resonator controlled via electrostatic actuation. This chapter provides a brief introduction into feedback cooling and serves to introduce the fundamental optomechanical system that underscores the majority of the work presented in this thesis. In the following chapter, the problem of estimating an unknown force driving a linear oscillator is considered. In this context it is well known that linear feedback control can improve the performance of nonlinear mechanical sensors. However, for completely linear systems, feedback is often cited as a mechanism to enhance bandwidth, sensitivity or resolution. For such systems it is shown that as long as the oscillator dynamics are known, there exists a real-time estimation strategy that reproduces the same measurement record as any arbitrary feedback protocol. Consequently, some form of nonlinearity is required in the controller, plant, or sensor, to gain any advantage beyond estimation alone. This result holds true in both quantum and classical systems, with non-stationary forces and feedback, and in the general case of non-Gaussian and correlated noise. As a proof-of-principle, a specific case of feedback enhanced incoherent force resolution is experimentally ...