Given R ⊆ N let n k R , n k R , and L(n, k) R count the number of ways of partitioning the set [n] := {1, 2, . . . , n} into k non-empty subsets, cycles and lists, respectively, with each block having cardinality in R. We refer to these as the R-restricted Stirling numbers of the second and first kind and the R-restricted Lah numbers, respectively. Note that the classical Stirling numbers of the second kind and first kind, and Lah numbers are n k = n k N , n k = n k N and L(n, k) = L(n, k) N , respectively.n,k≥1 and [L(n, k) R ] −1 n,k≥1 exist and have integer entries if and only if 1 ∈ R. We express each entry of each of these matrices as the difference between the cardinalities of two explicitly defined families of labeled forests. In particular the entries of [ n k [r] ] −1 n,k≥1 have combinatorial interpretations, affirmatively answering a question of Choi, Long, Ng and Smith from 2006.If we have 1, 2 ∈ R and if for all n ∈ R with n odd and n ≥ 3, we have n ± 1 ∈ R, we additionally show that each entry of [ n k R ] −1 n,k≥1 , [ n k R ] −1 n,k≥1 and [L(n, k) R ] −1 n,k≥1 is up to an explicit sign the cardinality of a single explicitly defined family of labeled forests. With R as before we also do the same for restriction sets of the form R(d) = {d(r−1)+1 : r ∈ R} for all d ≥ 1. Our results also provide combinatorial interpretations of the kth Whitney numbers of the first and second kinds of Π 1,d n , the poset of partitions of [n] that have each part size congruent to 1 mod d.