At present, most studies only conduct individual studies on distraction or fatigue, but only discussing these two accident factors cannot fully explain and avoid all car accident factors. In addition, most of the current research and application of physiological information belongs to the follow-up analysis, and the application of physiological information is not realized in real time, and there is also a lack of intelligent warning system integrating computer vision and physiological information. In order to solve this problem and increase the application of real-time practical and physiological information, this study uses YOLOv5 computer vision for detection. In addition to considering distraction and fatigue, YOLOv5 also includes emotion as a factor (Alkinani et al., 2020), and through the real-time auxiliary judgment of physiological information (EEG, HRV), an intelligent warning system is established to more comprehensively explain the factors of traffic accidents. It also addresses the lack of integration of distraction, fatigue, and emotion in previous driving safety research. However, there is currently a lack of public datasets that integrate distraction, fatigue, and emotion for use in the field of automotive safety. Therefore, this study also builds a well-generalized dataset for training in a driving simulation environment. In the research, YOLOv5 of deep learning is used to image the driver’s facial and body movement characteristics, and through the real-time processing of physiological information, it can assist in detecting driving emotions and driving fatigue, and establish a personalized intelligent driving warning system. A more practical vehicle warning system for the driver.