In recent years, a surging number of unmanned aerial vehicles (UAVs) are pervasively utilized in many areas. However, the increasing number of UAVs may cause privacy and security issues such as voyeurism and espionage. It is critical for individuals or organizations to manage their behaviors and proactively prevent the misbehaved invasion of unauthorized UAVs through effective anomaly detection. The UAV anomaly detection framework needs to cope with complex signals in the noisy-prone environments and to function with very limited labeled samples. This paper proposes BISSIAM, a novel framework that is capable of identifying UAV presence, types and operation modes. BISSIAM converts UAVs signals to bispectrum as the input and exploits a siamese network based contrastive learning model to learn the vector encoding. A sampling mechanism is proposed for optimizing the sample size involved in the model training whilst ensuring the model accuracy without compromising the training efficiency. Finally, we present a similarity-based fingerprint matching mechanism for detecting unseen UAVs without the need of retraining the whole model. Experiment results show that our approach outperforms other baselines and can reach 92.85% accuracy of UAV type detection in unsupervised learning scenarios. 91.4% accuracy can be achieved when BISSIAM is used for detecting the UAV type of the out-of-sample UAVs.