This article introduces an hp-adaptive multi-element stochastic collocation method, which additionally allows to re-use existing model evaluations during either h-or p-refinement. The collocation method is based on weighted Leja nodes. After h-refinement, local interpolations are stabilized by adding and sorting Leja nodes on each newly created sub-element in a hierarchical manner.For p-refinement, the local polynomial approximations are based on total-degree or dimension-adaptive bases. The method is applied in the context of forward and inverse uncertainty quantification to handle non-smooth or strongly localized response surfaces. The performance of the proposed method is assessed in several test cases, also in comparison to competing methods.