In this study we combine the expansion and the growth data in order to investigate the ability of the three most popular holographic dark energy models, namely event future horizon, Ricci scale and Granda-Oliveros IR cutoffs, to fit the data. Using a standard χ 2 minimization method we place tight constraints on the free parameters of the models. Based on the values of the Akaike and Bayesian information criteria we find that two out of three holographic dark energy models are disfavored by the data, because they predict a non-negligible amount of dark energy density at early enough times. Although the growth rate data are relatively consistent with the holographic dark energy models which are based on Ricci scale and Granda-Oliveros IR cutoffs, the combined analysis provides strong indications against these models. Finally, we find that the model for which the holographic dark energy is related with the future horizon is consistent with the combined observational data.