The Hubble tension seems to be a crisis with ∼ 5σ discrepancy between the most recent local distance ladder measurement from Type Ia supernovae calibrated by Cepheids and the global fitting constraint from the cosmic microwave background data. To narrow down the possible late-time solutions to the Hubble tension, we have used in a recent study [Phys. Rev. D 105 (2022) L021301] an improved inverse distance ladder method calibrated by the absolute measurements of the Hubble expansion rate at high redshifts from the cosmic chronometer data, and found no appealing evidence for new physics at the late time beyond the ΛCDM model characterized by a parametrization based on the cosmic age. In this paper, we further investigate the perspective of this improved inverse distance ladder method by including the late-time matter perturbation growth data. Independent of the dataset choices, model parametrizations, and diagnostic quantities (S8 and S12), the new physics at the late time beyond the ΛCDM model is strongly disfavoured so that the previous late-time no-go guide for the Hubble tension is further strengthened.