As computer systems become increasingly complicated, the number of vulnerability is increasing rapidly. This alarming trend threatens the safety of various applications. Especially it is very critical to security-sensitive applications containing intellectual property and confidential information. In this dissertation, we propose a virtualization-based approach to guarantee the trusted execution for highly secure applications.
The first part of our study deals with a new cloud marketplace model to support code and data protection at rest, in use, and in motion for a cloud computing environment. Cloud application market- places of modern cloud infrastructures offer a new software deployment model, integrated with the cloud environment in its configuration and policies. However, as the traditional software distribution is suffer- ing from software piracy and reverse engineering, cloud marketplaces face the same challenges that can deter the success of the evolving ecosystem of cloud software. We present a novel system named CAFE for cloud infrastructures where sensitive software logic can be executed with high secrecy protected from any piracy or reverse engineering attempts in a virtual machine even when its operating system kernel is compromised. The key mechanism is the end-to-end framework for the execution of applications, which consists of the secure encryption and distribution of confidential application binary files, and the runtime techniques to load, decrypt, and protect the program logic by isolating them from tenant virtual machines based on hypervisor-level techniques. We evaluate applications in several software categories that are commonly offered in cloud marketplaces. We show that strong confidential execution can be provided with only marginal changes (around 100-220 lines of code) and minimal performance overhead. The results demonstrate the effectiveness and practicality of CAFE in cloud marketplaces.