In recent years, with the popularization of network encryption technology, malicious attacks using network encryption technology have increased year by year. Traditional detection methods that rely on the content of data packets are now unable to effectively deal with malware attacks hidden in encrypted traffic. In order to deal with the detection of encrypted malicious traffic under different protocols, this study proposes an encrypted malicious traffic detection algorithm based on profile HMM. This method uses the genetic sequence comparison analysis in bioinformatics to realize the identification of encrypted attack traffic by matching key gene sub-sequences. Open source datasets are used to conduct experiments under different conditions, the results demonstrate the effectiveness of the algorithm. In addition, two methods of evasion detection are designed, and experiments have also verified that the algorithm has a better performance to resist evasion detection. Compared with the existing research, the work of this study has a wide range of application scenarios and higher detection accuracy. It provides a more effective solution to the research field of malware detection based on encrypted traffic.