The afterglow of a gamma ray burst (GRB) can give us valuable insight into the properties of its host galaxy. To correctly interpret the spectra of the afterglow we need to have a good understanding of the foreground interstellar medium (ISM) in our own Galaxy. The common practice to correct for the foreground is to use neutral hydrogen (H i) data from the Leiden/Argentina/Bonn (LAB) survey. However, the poor spatial resolution of the single dish data may have a significant effect on the derived column densities. To investigate this, we present new high-resolution H i observations with the Australia Telescope Compact Array (ATCA) towards 4 GRBs. We combine the interferometric ATCA data with single dish data from the Galactic All Sky Survey (GASS) and derive new Galactic H i column densities towards the GRBs. We use these new foreground column densities to fit the Swift XRT X-ray spectra and calculate new intrinsic hydrogen column density values for the GRB host galaxies. We find that the new ATCA data shows higher Galactic H i column densities compared to the previous single dish data, which results in lower intrinsic column densities for the hosts. We investigate the line of sight optical depth near the GRBs and find that it may not be negligible towards one of the GRBs, which indicates that the intrinsic hydrogen column density of its host galaxy may be even lower. In addition, we compare our results to column densities derived from far-infrared data and find a reasonable agreement with the H i data.