Driver stress detection has great potential for implementing assisted driving because the stress of the people is closely related to their behavior, especially in smart driving. The existing stress perception methods are often used in static environments and lack of convenience, so it is difficult to satisfy the highly dynamic smart driving environments. This study proposes a behavior-assisted stress perception method based on wearable system to achieve natural, accurate, and reliable stress detection in smart driving. This method based on the behavior and multiple metrics to distinguish stress state, can effectively improve the stress detection accuracy. The basic principle is that each person's physiological characteristics and behavioral habits under different stress conditions will have unique effects on stress-related PPG data and behavior-related IMU data. The driver's physiology and motion information are measured using a multi-sensor wearable glove, and then reliable physiological and behavior metrics are obtained through multi-signal fusion techniques. Finally, the SVM model is used to classify the driver's stress state because of good generalization performance. Based on the proposed method, this study deploys a verification experiment in a simulated driving environment, the experimental results show that the stress classification accuracy can reach 95%.