The interaction between plants and microorganisms, which is the driving force behind the decontamination of petroleum hydrocarbon (PHC) contamination in phytoremediation technology, is poorly understood. Here, we aimed at characterizing the variations between plant compartments in the microbiome of two willow cultivars growing in contaminated soils. A field experiment was set-up at a former petrochemical plant in Canada and after two growing seasons, bulk soil, rhizosphere soil, roots, and stems samples of two willow cultivars (Salix purpurea cv. FishCreek, and Salix miyabeana cv. SX67) growing at three PHC contamination concentrations were taken. DNA was extracted and bacterial 16S rRNA gene and fungal internal transcribed spacer (ITS) regions were amplified and sequenced using an Ion Torrent Personal Genome Machine (PGM). Following multivariate statistical analyses, the level of PHC-contamination appeared as the primary factor influencing the willow microbiome with compartment-specific effects, with significant differences between the responses of bacterial, and fungal communities. Increasing PHC contamination levels resulted in shifts in the microbiome composition, favoring putative hydrocarbon degraders, and microorganisms previously reported as associated with plant health. These shifts were less drastic in the rhizosphere, root, and stem tissues as compared to bulk soil, probably because the willows provided a more controlled environment, and thus, protected microbial communities against increasing contamination levels. Insights from this study will help to devise optimal plant microbiomes for increasing the efficiency of phytoremediation technology.