To detect illegal behaviors, cameras have to be mounted in public space to analyze human behavior to determine whether it is illegal. In this thesis, we focus on detecting smoking and drinking in certain public spaces as the illegal behaviors. Comparing with other works, our work does not need to establish the background in advance to classify human behavior.The proposed system consists of four modules including face region extraction, multiple hand samples extraction, features extraction, and behavior analysis. We extract three features from each human behavior. They are the touching time between the face and hand samples, smoke detection, and handheld object detection. Then a decision tree is employed to classify the human behavior by using the extracted three features. Experimental result demonstrates that the proposed method can be suited and successfully applied in many environments under various conditions, such as different illumination intensities, different backgrounds, and the different habits exhibited by human.