All persons in self-driving vehicle would like to receive each service. To do it, the system has to know the person’s state from emotion or stress, and to know the person’s state, it has to catch by analyzing the person’s bio-information. In this paper, we propose a system for inferring emotion using EEG, pulse, blood pressure (systolic and diastolic blood pressure) of user, and recommending color and music according to emotional state of user for a user service in self-driving vehicle. The proposed system is designed to classify the four emotional information (stability, relaxation, tension, and excitement) by using EEG data to infer and classify emotional state according to user’s stress. SVM algorithm was used to classify bio information according to stress index using brain wave data of the fuzzy control system, pulse, and blood pressure data. When 80% of data were learned according to the ratio of training data by using the SVM algorithm to classify the EEG, blood pressure, and pulse rate databased on the biometric emotion information, the highest performance of 86.1% was shown. The bio-information classification system based on the stress index proposed in this paper will help to study the interaction between human and computer (HCI) in the 4th Industrial Revolution by classifying emotional color and emotional sound according to the emotion of the user it is expected.