Investigation of deep transfer learning on machinery fault diagnosis is helpful to overcome the limitations of a large volume of training data, and accelerate the practical applications of diagnostic algorithms. However, previous reported methods, mainly including parameter transfer and domain adaptation, still require a few labeled or massive unlabeled fault samples, which are not always available. In general, only extremely limited fault data, namely sparse data (single or several samples), can be obtained, and the labeling is also easy to be processed. This paper presents a novel framework for disposing the problem of transfer diagnosis with sparse target data. In consideration of the unclear data distribution described by the sparse data, the main idea is to pair the source and target data with the same machine condition and conduct individual domain adaptation so as to alleviate the lack of target data, diminish the distribution discrepancy as well as avoid negative transfer. More impressive, the issue of label space mismatching can be appropriately addressed in our network. The extensive experiments on two case studies are used to verify the proposed method. Comprehensive transfer scenarios, i.e., diverse working conditions and diverse machines, are considered. The thorough evaluation shows that the proposed method presents superior performance with respect to traditional transfer learning methods.