Abstract:Cloud computing is gaining momentum against traditional method in providing users various services with greater flexibility and scalability. Before switching to cloud computing, users must take into account the security of cloud as an extremely important factor. That is because in the cloud environment, attackers can initiate efficient attacks to cloud users through the shared cloud resources such as virtual machines. Since virtual machines (VM) are basic resources of cloud service, by compromising or renting several virtual machines, attackers may deploy malicious software into those machines and launch a wider range of attacks to other virtual machines such as distributed denial of service (DDoS). To tackle this issue, this paper proposes a defense in depth system based on software defined networking to be able to detect suspicious virtual machines and monitor the flow they issued in time, and inhibit the aggressive behavior from the suspected virtual machines to mitigate the attack consequences. The system detects the virtual machines' running state in a completely non-intrusive and agent-free way, and monitors network traffic between virtual machines on the same host or between cloud hosts at process level based on software defined networking. Experimental results demonstrate the effectiveness of the system.