Conversation conveys important social signals of human interaction that indicates interest, service-awareness, persuasiveness, etc. In this paper, the authors employ the most common setting of using microphones to capture spontaneous conversation, and introduce a privacy-preserving and languageindependent speech processing approach that can detect speaking and separate speakers in high accuracy for such setting. Experimental results have validated that the approach can deliver accurate speaking recognition results in Japanese, English and Chinese conversation, and can be processed in real time applications.