Live migration of virtual machines is the transfer of running virtual machines from one host server to a new host server to ensure computing tasks completed without notifying the owners of virtual machines, which has many beneficial characteristics such as load balancing, hardware independent, and high efficiency utilization of resource. However, live migration of virtual machines exposes information of virtual machines and their users to the network, making its security in the virtualized environment a serious problem that concerns many users becomes a hot issue in the industry and academia. This article focuses on researching the mechanism of virtualization and the source code of virtualization operating system, and explores breakthrough in security problems of live migration. Firstly the article analyzes potential memory-leak security threat of live migration. Then it designs and puts forward a new security protection model based on hybrid random transform coding method. Combined with KVM (kernel-based virtual machine) virtualization structure, communication mechanism and migration mechanism, the model adds monitor module and security module at source and destination of live migration, ensuring the data security while the virtual machines are migrating. Finally, a series of experiments are designed to simulate and test the security protection capability of the model and its impact to virtual machine's performance. The simulation results show that the proposed model can ensure the security of live migration in the KVM virtualization environment, as well as balance the security of virtual machines and performance of live migration.