SPam over Internet Telephony is a major issue in Voice over Internet Protocol (VoIP) systems where a large number of automated unsolicited calls are made to users of VoIP. The economy of communication that VoIP brings is a lucrative proposition to spammers. In this paper, we describe a method to detect spammers in VoIP by identifying anomalies in the CallGraph. This directed, weighted graph is generated using call data records of users where a set of differentiating call parameters are used to derive weights on the edges. We identify anomalies in the graph by considering the local neighborhood of a node under consideration and assign a label based on how similar the node is in comparison to its neighbors. The similarity between a node and its neighbor is measured through a parameter known as SpamOutlierFactor. We experiment with a large simulated user base and show that the proposed method can detect spam callers.