The successful application of deep learning in bridge damage diagnosis relies on the assumption that the training and test data sets obey the same distribution. However, it is difficult to obtain labeled data of damage status for a bridge in using. Otherwise, it is difficult to apply a model trained with bridge A (source domain) to diagnose bridge B (target domain) because of the distribution discrepancy of data from different working environments or bridges. In response to these problems, motivated by transfer learning, a new bridge damage diagnosis method, namely, the multichannel domain adaptation deep transfer learning based method (MDADTL), is proposed in this paper. First, a CNN based multichannel multi-scale feature extractor is introduced to extract features. Second, a multichannel domain adaptation module based on maximum mean discrepancy (MMD) is proposed for transfer learning, so that the learned features are domain-invariant. Through the above process, MDADTL trained with labeled data obtained in the laboratory or the testing bridge is expected to diagnose other bridges with unlabeled data. Experiments prove the effectiveness and advancement of the proposed method. This exploration will promote the practical application of deep learning in bridge damage diagnosis.