Abstract:This paper presents AppISO, a novel approach to provide whole-application protection in an untrusted operating system(OS). Unlike previous virtualization-based approach, AppISO does not directly use any higher privilege hypervisor for application protection, which is known to cause high overhead due to frequent privilege transitions. Instead, AppISO introduces a software component named Inner TCB running in the same privilege layer with the untrusted OS, and uses Inner TCB to realize application protection. Meanwhile AppISO leverages hardware virtualization and software techniques such as page table lockdown, shadow IDT, and transition page to guarantee the security and isolation of Inner TCB. This paper proves that Inner TCB can achieve the same level of security as hypervisor, and experimental results show that the presented approach has significant improvement in performance.