As a security defense technique to protect the network from attacks, the system of network intrusion detection system, as a security defense technology to protect the network from attacks, plays a very important crucial role in the field of guaranteeing computer system and network security. However, for the multi-classification problem of unbalanced data in network intrusion detection data, machine learning has been widely used in intrusion detection so as to achieve high intelligence and accuracy. In this paper, the current multi-classification method for network intrusion detection is improved, and an intrusion detection model RF-GBDT is proposed, which applies based on the random forest model for to feature conversion and classification using the model of gradient boosting decision tree to classification model is proposed. The model is mainly includes divided into three parts:Feature selection, feature conversion, and classifier. The UNSW-NB15 dataset was used for the experimental data set to test; experimental tests were carried out on the RF-GBDT model. Compared with the other three algorithms in the same field, RF-GBDT, this model not only reduces training time, but also has a higher detection rate and a lower false alarm rate. The area under the subject's working characteristic curve on the test data set can reach 98.57%. RF-GBDT, the proposed model has significant advantages in solving the multi-class problem of multi-classification of unbalanced data in network intrusion detection data and is a feasible method for network intrusion detection.