The Impulse-Response (IR) test is the most commonly used field procedure for assessing the structural integrity of piles embedded in soil. The IR test uses the response of the pile to waves induced by an impulse load applied at the pile head in order to assess the condition of the pile. However, due to the contact between the pile and the soil, the recorded response at the pile head carries information not only about the pile, but about the soil as well, thus creating the, as yet unexplored, opportunity to characterize the properties of the surrounding soil. In effect, such dual use of the IR test data renders piles into probes for characterizing the near-surface soil deposits and/or soil erosion along the pile-soil interface.In this article, we discuss a systematic full-waveform-based inversion methodology that allows imaging of the soil surrounding a pile using conventional IR test data. We adopt a heterogeneous Winkler model to account for the effect of the soil on the pile's response, while the pile's end is assumed to be elastically supported, thus also accounting for the underlying soil. We appeal to a PDE-constrained-optimization approach, where we seek to minimize the misfit between the recorded time-domain response at the pile head (the IR data), and the response due to trial distributions of the spatially-varying soil stiffness, subject to the coupled pile-soil wave propagation physics. We report numerical experiments involving layered soil profiles for piles founded on either soft or stiff soil, where the inversion methodology successfully characterizes the soil.