Human activity recognition (HAR) aims to recognize the activities of daily living by utilizing the sensors attached to different body parts. The great success of HAR has been achieved by training machine learning models using sufficient labeled activity data. However, when the labeled data from a certain body position (i.e. target domain) is missing, how to leverage the data from other positions (i.e. source domain) to help recognize the activities of this position? This problem can be divided into two steps. Firstly, when there are several source domains available, it is often difficult to select the most similar source domain to the target domain. Secondly, with the selected source domain, we need to perform accurate knowledge transfer between domains in order to recognize the activities on the target domain. Existing methods only learn the global distance between domains while ignoring the local property. In this paper, we propose a Stratified Transfer Learning (STL) framework to perform both source domain selection and activity transfer. STL is based on our proposed Stratified distance to capture the local property of domains. STL consists of two components: 1) Stratified Domain Selection (STL-SDS), which can select the most similar source domain to the target domain; and 2) Stratified Activity Transfer (STL-SAT), which is able to perform accurate knowledge transfer. Extensive experiments on three public activity recognition datasets demonstrate the superiority of STL. Furthermore, we extensively investigate the performance of transfer learning across different degrees of similarities and activity levels between domains. We also discuss the potential applications of STL in other fields of pervasive computing for future research.